Lulu is by far the one who's lost most in this. Also his little sister... He needs to bring them with him and spoil them for awhile to make up for all those years of being an asshole.
I don't believe he ever was an asshole. His behavior and countenance were just unbecoming to a prince, but not necessarily in an assholish way. The little sister always loved him and he spoiled her. It was being obese and terrible physically, failing at the dragon knight academy to the point of getting expelled, and acquiring the reputation for being a peeping tom. Well, that's plenty, but he apparently wasn't an asshole. After all, he even kept donating a lot of money (that he earned himself with alchemy, to boot) to the temple for charity purposes. The only time he was rude to Lulu was not really intentional. It was when he told her to find a better man, not a failure of a prince.