He's a bullying jerk, but he comes across as really not knowing how much she was abused, and it's stated that he does this because Yvonne is still missing and looking at Penelope just reminds him of his missing sister that could still be in danger, he does do bad stuff, but once he sees the food she was served and really gets a feel as to how she was treated, he starts to realize the reason why she acted out and is the first one to mellow, and no, Penelope doesn't forgive him or want anything to do with him once he starts getting nicer.
In fact, his interest rises precisely because he sees that Penelope actually had a bad life and realizes he was a jerk, he is also pratically the only love interest of the original game that doesn't fall for Penelope...yes the older brother that attacked Penelope for causing problems to the maid that gave her trash food will fall in love for the main character, but not this guy, at worst he just starts considering her a sister by the end, so at least we don't have to read any romantic scenes from this guy.
But yeah, even if Penelope understands he's getting better and some of his reasons for acting out, she doesn't become a fan and actively tries to avoid spending time with him once he starts trying to hang out with her, and yes, it's heavily implied the guy didn't know that the servants and the rest of his family were abusing Penelope when he wasn't watching or behind it, he actually goes on to try being around Penelope to be her guard or make others less willing to abuse her, but this is once he starts getting a feel as to how bad her life is and how wrong he was.