Elinalise ought to respect Rudeus a bit more when he didn't immediately jump into marrying Roxy as well. It shows he's still dedicated to his first wife, who just happens to be Elinalise's grandchild.
I have to say this is the kind of harem building I can respect, instead of the romcom type in 99% of harem titles. The other acceptable type (in my eyes) is the political one, as long as the story is otherwise convincingly political, so that it really seems like every new member is essential for greater reasons than just to make harem loving readers happy.