its not that he cant take his bracers off, its just that he doesnt want to take them off.
presumedly the logic is along the lines of "if i can beat him with them on despite him being legitimately strong, i'll get even stronger", and so he wont take them off until he absolutely needs to in order to win.