Honestly I wouldn’t even give him the benefit of even that minuscule level of (manufactured) attraction and empathy. Based on everything we know about him so far it’s clear he never actually loved her and just saw her, or rather the happiness that she exuded as a “kid with nothing” like himself, as something to be acquired and treated like a toy or heirloom.
The key difference between Yamato and uishi is that Yamato never reached out, not just to FMC but to anyone, did he ever invite her out to play soccer? Did he see her off when she transferred schools? Did he try to keep in contact at all? Did he ever even ask uishi if he likes her? No, he just sat on his hands and swatted away any opportunities to branch out because he was pompous enough to think, even as a little kid, that fmc would return to him like a dog to its rightful owner and later that uichi’s life was animal control picking up a supposed stray to take to the veterinarians office.