@ShionVaynex wasn't he stripped of knighthood and his title? At this point he could marry Olivia. On the other hand, given his power, money and involvement, I think her father wouldn't mind giving him Angelica...
@chilzhul at least we could distinguish between the cookie-cutter harem members during naked time... I had to reread the bath scene two times to understand who is saying what.
@Izaya_Onisai he feeled responsible for her, since she was supposed to get his ship; when miss trollface appeared and went for the harem route, he befriended her in order to help and get insight into what was happening. If you reread chapter 5 it seems he actually care for her, while Angelica is only a step stone in his master plan.