@monkey123 Realistically dragging non combatants around is indeed a burden. You would need to leave her somewhere safe and constantly think about her safety or limits. He literally just said this chapter that he couldn't protect the Dwarf. If you carry around other people it slows you down. If she was a fighter it would be a different matter of course. A merchant however is just as useful to him back in town as she is following him around.
His portals also don't have unlimited range. So regardless he is going to have to leave her behind at some point. If he wants a Harem member to bang he needs to find someone who can actively contribute to his journey. He has a book of all knowledge so that essentially means he only needs combatants. Rogues, warriors, other mages, healers etc.
@ShadowWalker Remember the mine? He did that because the reward was the hammer.