There are several things confusing and frankly irritating here:
1. Why did the first prince annulled the engagement? the FL did not do anything. If the reason is simply "he did not loves her", then I hope he have to pay enormous alimony to FL and also get thrown out of Crown Prince candidacy because seriously, he is part of the royal family, which means ARRANGED MARRIAGE IS A NATURAL THING and he's a dick for just annulling the engagement nilly willy.
2. Did the first prince actually going out of his way to annul the engagement in the fourth prince's island? Seriously?? That is taking being a bastard to another level. He knows he can just came to her house or called her to the castle right?
3. Why did the Fourth Prince get called as a 'notorious noble'? so much that other nobles called out "how dare such insignificant thing like him came to the party."
4. Idiotic nobles. We still dunno about Fourth Prince's background, but guessing how much he is get belittled, I'm guessing his mother is maid or a commoner or something, but seriously, hello guys, he is STILL A PRINCE. Which means he has the KING'S BLOOD within him. From that alone you guys did not have enough status in you to belittle him in anyway. Did they actually want to commit Lèse-majesté???