Why the heck would the noble girl (the hero's fiance) and the entire harem want to go see Sansui's master? Just because the hero is going? Such clinginess! Learn about personal space!
And Sansui say it's okay? Has Sansui forgotten that his teacher is a hermit? You know, the type of people who say, "Get out of my lawn, damned kids!"