I mean the girls already said that they would watch-over-him from afar/distance in the Manga.
Aka., they have no will to even connect to him emotionally in the Manga, let alone any physical interaction.
Atleast the Dungeon Emperors are trying to connect to him in someway and stay close to him...