The pace in the early chapters is wacky, and I'm not sure if it's due to the translation or what, but the dialogue was hard to follow. So, yeah, he was being a hikikomori seems to come out of nowhere, although I know he was always socially awkward.
Maybe because she is the queen and it is her Garden, so she can make the rules; she wanted to have her closest friends as aides, so be it.
In short, probably different factions, different queens, different rules