The reason (ostensibly--the actual reason, as always, could be whatever the author wants it to be) Serena fails to communicate effectively with her mother Amaryllis is the fact that Serena does not, according to what's been shown, communicate or even think like a woman: Serena does these things logically (having had all her femininity beaten out of her in her previous life), which is an inherently masculine way of doing things, while Amaryllis functions almost totally (if not totally) on emotion--an inherently feminine MO.
As a result, Serena's logical appeals don't actually reach Amaryllis--who, at best, only attempts to allay Serena's concerns with attempts to end any given conflict in general (i.e., as long as the person she cannot relate with/does not feel positively toward is in the right; a characteristically feminine method of pursuing community harmony as opposed to actual conflict resolution). This sort of mentality makes Amaryllis a perfect mark for Rosemary, who argues only with (often highly emotionally-charged) emotional appeals.
Rosemary speaks Amaryllis' language, while Serena does not; it's no wonder there's such a divide between Serena and her mother--but not between Amaryllis and Rosemary.