Well, I feel bad for the girls in the prequel story for this. They never got him to love them but now, their successors get the chance instead of them.
@SUBARU I don't, they all knew his circumstances and still went along with the "confinement" plan, then they choose an inept to convey it to him. As much as the MC doesn't have any resentment (of course since it's a shounen), they don't really have any foundation to be emphatized with.