I mean lets be honest. What is there to like about him outside of his fantasy tier stamina. No looks, no academics, almost monkey tier IQ and his dick takes the wheel %99 of the time. If anyhing, Agawa should end up with him to save all the other girls any possible future pain of ending up with him. They deserve each other anyway as Agawa also doesn't have anything to offer other than her VERY high mileage pussy.
Maybe I will sound like a Gotou glaser, but to me it makes sense.
True he isn't the most handsome, but no academics?, sure he failed his mid semester exams, but he also got a 100 when he took those tests again, if he locks in bro has academics. Also true he lets his dick take the wheel, but honestly the only one that was bothered by that was Agawa until resent chapters, the rest either like it or don't seem to mind.
However to me each girl has a reason to like him but honestly you are right with the stamina, it is a factor but it isn't the whole picture.
Agawa & Shigemi: To me this 2 like Gotou for the same reason, Gotou has never treat them as sluts, he respects them as people and really important especialy for Shigemi, Gotou respects their boundries ( In fact when Gotou respects Shigemi's boundries, she always says "what is wrong with you", to me this is really telling about how other dudes behave with her). If you look at early chapters and see how some guys call Agawa then you realize why her standards are really low for liking a dude, also lets not forget that Gotou and Agawa have history from before the begining of the manga, for a while Gotou was Agawa's only male friend the rest were sexfriends.
Shiina: To me Gotou earned her respect in the summer arc by taking his studies seriouly and listening to her advice, considering that she went from trying to get him as far away from Agawa to offer him an expensive ED treatment and trying to protect him from Kiyomi, Gotou is probably the only man that Shiina sees as more than a sex toy. Also she likes that he is an idiot aparently.
Akito: Because childhood friend, a little basic but she has known Gotou for a while and likes his kind nature, also she admired him as a baseballer specially when he got serious.
Kiyomi: She probably is the one that really cares about his stamina, but also they have history together so that counts for something i guess.
tl;dr Gotou is a huge idiot but is the only one that trats these girls with and ounce of respect.