See, this is just stupid reasoning from Tsukasa's older brother. He's going by the old mentality that the sins of the father are the sins of the sons, etc. Ryuji himself hasn't harmed female MC, in fact, he has saved her plenty of times when she herself got into trouble. W/o male MC, female MC would have drowned on two separate occasions.