Could it come from the *suspicious* novels he read on romance before coming to the estate? I think that would explain the shoujo-mindset which led to this weird conclusion.
I must admit also that author really has a talent for double sensed conversations. 'tis a common trope to make a kind character misunderstood due to his cold/intimidating appearance or demeanor, but here it's on a really another level, with all the details and the quiproquos adding on to another without ever getting dramatic...