Thanks for the translation!
I don't have a difficult time at all believing that someone who sleeps around freely has slept with people they don't remind the faces or names of afterwards and I don't think that's a problem.
To me, the problem writing-wise with trying to make him the father (if the mangaka has/had any desire to do that) would be that the flashback conversation with the landlady showed that she and the father spoke after they had sex and he knew she was pregnant. I would have a hard time suspending my disbelief towards the idea that he still wouldn't remember the face or name of a woman that he had that degree of interaction with.