When did Haru pledge herself to such a weakling? First he stands back and watches when Haru is locked in a difficult fight, now he's squirming in front of a physical attacker, squealing about primarily using magic? At least where anything involving combat is concerned, the author is falling into the same habits of his peers: thinking an effeminate male who waves his hands and makes spectacular things happen--while hiding behind women doing the physical combat he should be capably engaging in--makes for a strong male protagonist.
I could understand if he had a reason to avoid engaging in physical combat, but it's not his Jobless powers he'd be using nor would great swordsmanship easily be identified as something so extraordinary as to require the kind of scrutiny he wants to avoid; even if he was scrutinized, there was no way to trace the physicality he used in the beginning of the story to his Joblessness--and if he were to use it now, he could just explain it away as diligent training.
I really thought I was wrong, with how Ichinojou fought in the previous chapter.