Yeah, honestly, in the real world, there are consequences to wearing clothes. It's a cool line in a vacuum(the "men wearing women clothes/vice-versa" line), but some people can't handle the scrutiny.
That's not to say you can't do what you want, because you absolutely can. And if its not...