I guess he was thinking that a hakama is unisex, and he could wrap up and hide the chest, but it hurt too much?
Also, if they're still focusing on flags, why didn't they go wearing normal boy clothes? That's what usually happens in these stories, right?