If you read the light novel, it mentions that he replaces his clothes regularly. It's just that he only knows how to make rope sandals and a simple robe.
Also, when he takes service with Dewey, she replaces his clothes with identical ones... just well made using the best cloth. Oh, and he...