I think it is older female and younger male that might be making people a little uncomfortable. Nagisa-san seems like a working adult while the younger brother looks like a college student (or even high school. It’s hard to tell, but it is quite clear that he is younger than Nagisa-san)
I don’t mind the fact that the main couple are related to this second couple. I think that’s kind of fun.
I am also looking forward to the main couple having kids. I will riot if they do not at least show the kids by the end of the manga serialization.