@HiraOmnibus and others I will try a last time explaining the MC. He is basically traumatized and in some way not right in the head because of that. that hasnt anything to do with self esteem or whatsoever and is nothing that can be that easely improved, especially if he lived that long with it already. Because of his former life it became something of his character and self. He will need to work hard on himself to improve and maybe be able to let it behind him.
That he is too hard on himself is obvious for every outside person who doesnt have this trauma, but people like him see that differently. Normally Ryouma would need a psychologist or something like that. Sadly this world doesnt have such a thing. Therefore riding the wave that he is wrong is pointless, because the MC isnt able to snap with his finger and see that he was in the right. Heck he even knows it himself logically but emotionally he cant accept it. That is what can be called a disorder when it is this extreme.
Another point with that she cant see him for 3 years .... aristrocratic schools are most of the time at the capital or something, also quite the distance away. Visiting isnt that easy and at best Ryouma could try visiting them if he has time, but because of his personality it would be strange to do it, if he isnt accidently in the capital.