This dude is making a harem of boys and girls 🤣
He clearly does not want any of them in that way, it's just a group of friends. I know this "friends" thing might sound absurd for some manga (especially being an isekai, but do take notice that there's no haren tag, which is rare but a plus for me), and even more so for some people that read manga, but I've read about it in other manga and this "friendship" thing might actually be true.
"I don't want to get involved in the story..."
Proceeds to coerce the Female Lead into going home with him
This is more like he doesn't want to follow the story path, in which he wasn't involved at all, thus, getting himself involved and changing everything is what he's aiming to: in the "original story" she was supposed to stay there, so he will take her away from there even if he has to pay her to do so (might even be a reason for another prince to visit his land - for "sightseeing", thus promoting it even more).