@Panino Poor people definitely went around half-naked a lot of the time. When you only own the one set of clothes, there's not a lot of opportunity to wash it, so you'll try and keep it in decent condition by not wearing it while doing sweaty/smelly labor. That's a reason why tattooing was so popular until it was banned, it was a way of "wearing" clothing. It wasn't until the Meiji government tried to modernize/Westernize the country that nakedness became frowned upon.
"The houses were alive with unclothed children, and as I repassed in the evening unclothed men and women, nude to their waists, were sitting outside their dwellings with the small [children, who were] clothed only in amulets..."
"[Children] don’t wear any clothing till they are seven or eight years old, and are then dressed like their elders."
"Few of the men wore anything but the [loincloth], the women were unclothed to their waists and such clothing as they had was very dirty, and held together by mere force of habit."
That's not to say quite a few instances aren't simply fanservice though...
Incidentally, since Unbeaten Tracks in Japan is in public domain, you can
read it for free online here or the Kindle store has a free copy.