You cannot logically complain about harem in a work that pretty much by definition was going to have polygamy from the start.
Its like what the childhood friend that makes a bento said, once you cook for more than one person, at that point, 2, 3, 10 doesn't matter. Once you got more than one girl, at that point whether its 3 or 10 , eh.
Literally the chapter before this the mother was already planning to match him with some branch and vassal members in addition to daughters. Only shock is he manned up and basically accepted the harem from the start.