Pretty sure that was their point, that at least he's only dealing with one loud guy instead of being surrounded by a bunch of people reading while forming all these thoughts/mental noise.
It's because two panels before that Confucius was wondering if they stumbled upon a ghost market. The seller offering them samples of something that sounds like it comes from a literal ghost market made him basically exclaim "So it is a ghost market!"