All the characters have some kind of hangup, including the girls. Inuyama is actually a really well done character. He's the inverse of Seto. Inuyama is very confident and honest about what he's thinking, and he is also really self-aware of his own actions. A good example is in ch59 when he thought, "I think I've butted in too much," but he does it a lot. His major problem is he is over confident about his own unlikability, and so he overplays it as an immature defense mechanism.
I like how he slots in with the others. It's nice to see a pervy character who is not one dimensional, and not just there for comic relief fodder like in most manga.