On the one hand, Azami had sex with Mamoru, then chose to keep the baby (with Azuma's consent, as selfless as it may have been), and screwed with the family as a (perhaps unforeseen) result; Manabu was then forced to live in the annex till he parted with gramps. Only after they left did she come up with the idiotic plan to have Manabu hate her so he could live his own life outside the main house — as if talking things out and stating she did not want him to live the same life as Mayami was such a problem. I mean, sure, she thought he already hated her, but come on.
On the other hand, Mamoru found out his sister finally divorced her husband. As for what kind of “love” we're talking about here, I'm willing to assume it's not “just” the bond between brother and sister, given the contents of the letter and his whole charade. At any rate, his true love being forbidden — and once presumed unattainable — does not allow him to swindle her for extended amounts of time (if we can count the gossiping women as evidence), knowing full well what he's doing. Not to mention his total lack of interest in what is effectively his son, despite Gramps' two calls and despite having no unfortunate family situation onto which to blame his utter neglect — or for that matter his contemptible lack of interest in anything regarding the issue.
Given this, I contend the biggest asshole here is Mamoru.