Maybe the brother likes them because they have childhood and child's bodies, and he never had either?
I don't know, I just don't want to the brother to be two-faced or manipulative. The twins never had a place to belong or where people being friendly. Apparently all royal maids are shit, huh...