I bet, because this is Japan we're talking about, the more he increases his charm skill, the more he looks like an "Ugly Bastard" to others. Because, for some reason, only the ugliest motherfuckers in the world have sex with beautiful women in like 86% of manga nowadays.