It looks completely brickheaded in English, but there's some tricky Japanese ambiguity in play here: he used the word 付き合う to mean "go out with", but Mr. Oblivious took it as its other meaning "to associate with". Not entirely unreasonable, considering the only two "relations" he has with...