Men also wear garter belt, back then and even today too.
Back then they use it to hold the sock upright, because socks were not elastic like today socks.
They also wear it to keep shirt stay tucked in. The Marines still using it when they wear Brave or Charlie service uniform.
https://terminallance.com/wp-content/uploads/2012/05/military-shirt-stays.jpg