Imagine falling in love with some random kid you met for a few seconds once, and then no other person ever approached you, or made you like them or love them since. They aren't even trying anymore lmao.
Also, as other mentioned, he spent years in the other world, so, that might explain why he promised to pay back the favor, but forgot.
And also, he has never confessed to any of her fiances, they were the ones going for him, and more often than not, he can't really deny them, he hinted at it with the first two and both of their parents almost killed the kid with their glares. I know this is an isekai harem and all that, but, they should at least try a little smh.