I rather side with him-- there's just tremendous variability to even 'standard' women's clothes that men's tend to lack. Compounding it is that men's clothes tend to have distinct names, whereas women's have modifiers. (Example: 'Shirt' immediately makes general, but within a fairly narrow range, statements as to material, style, weight, formality, etc. 'Polo' gives an entirely different range. Unspecified 'dress'? I'm sure somebody pictured a wedding, and somebody else pictured a nightclub... a rather broad field.)
I used to work for an online fashion retailer and it was the biggest headache ever describing women's clothes... Things would end up with like 30 descriptors ("high-cut empire-waisted split one-piece with teardrop decollete") and it still wouldn't cover everything.
Men's? "It's a flannel polo shirt in grey/blue plaid."