Remember that in the beginning he trusted someone (the bitch princess) and got fucked sideways; my boy has PTSD and the slave crest is his only way to have mental stability.
In other isekai the mcs have slaves "just because" but here it's more of his character flaw.