I think the problem these people are having with him is that his reasoning isn't out of self-concern, but out of concern for them. When Kanda and Yoshida were dating, he wanted to wear a condom before sex, not because he's more comfortable wearing one than not wearing one, but because he wants to treasure Kanda's body. He's deciding for her that a condom would be better. I get it sounds messed up, because it's ultimately going on his penis, but his motivations are out of concern for her, when that's ultimately her call.
A better example is when Kanda offers to have sex with him after their reunion. He doesn't decline because he's not interested in sex with her (as far as we can tell, granted he doesn't outright confirm he is interested). He declines because she should treasure her body more. It's patronising. Another example is when he refuses to have sex with Sayu. He confirms that he is aroused when she goes out of her way to present herself sexually to him, but still he doesn't want to have sex with her, because he believes he'd be doing her a disservice.
Yoshida doesn't take his own desires into account when it comes to intimacy, he merely decides that others have made a mistake if they want to get intimate with him. I like how this chapter offers us another angle at this, which is that he has low self-esteem, as you mentioned.
You're totally right. But that conflict is what I meant when I said that "using your own agency always denies somebody else the use of their own agency, unless you both agree."
Your point about how Yoshida doesn't take his own desires into account is very astute and accurate, and you clearly grasp why they're upset with him. However, I feel it's still missing an important part of the picture- in that the desire to protect them is
also his desire. Humans are complicated, and his desire to protect those around him and keep them from harm is also real - and evidently more important to him than his desire for sex. I don't think it's entirely fair to resent him for being who he is. If he has bad chemistry with lovers like Kanda, he has bad chemistry - but that's not necessarily a personal failing. I think it's being presented that way in a narrative sense, but at least for now, I disagree with that message.
Let's take a more extreme example, just for illustrative purposes. If my girlfriend told me that she wanted me to beat the shit out of her during sex, I really wouldn't be comfortable with that. I may be a top, and I may even be a bit of a sadist, but I could never fathom
actually hurting the person I love. There may be a dark part of me which
could find joy in that - after all, many humans do find joy in extreme BDSM, and I too am human. Surely I have the capacity to enjoy hurting someone who wants to be hurt. But I'm not comfortable feeding or nurturing that part of me. I've had to use violence before, and I never want to turn that force against someone I love.
Yoshida's dilemmas are, admittedly, much less extreme. But the point is that we all have boundaries, and his boundaries are his to draw. Resenting him for that is unkind. It's okay to be uncomfortable with casual sex. It's okay to be uncomfortable with taking advantage of a highschooler. It's okay to be uncomfortable with raw sex if you're not married or trying for kids. What's important is that you and your partner have an understanding of each other's boundaries. Yoshida's problem is that he and Kanda never came to that understanding.
I know that may sound contrary to what I posted before, but I don't feel it is. We should strive to grow beyond boundaries that
we ourselves feel hold us back, and we can only do it at our own pace. Some boundaries are flexible. Some are not. And if they're going to change, they're going to change at the pace we're comfortable with. I think Yoshida would benefit more from encouragement and help, as opposed the haranguing that he's gotten. He's obviously a gentle and sensitive guy. Chastising him is only going to feed his feelings of inadequacy.
That's just me though. I appreciate that the story is emotionally complex enough to spark these sorts of serious conversations about the nature of relationships.
Edit: Put more simply, the decision he made "for Kanda" was in truth made for his own conscience, and is a product of his own nature. Whether he feels that boundary needs to change for the sake of his relationship, or whether he feels they were simply a poor match, is up to him. His decisions should be respected, rather than resented, and he ought to do the same for others. As should we all.