He's also dangerously incompetent and a total pain in the ass, besides having offended the girls on multiple occasions. The party wants nothing to do with him after the trouble he caused them last time.
Him being a noble with no power or know-how makes it dangerous for them to get involved...