You know, the guy can't hold a sword, but he's basically IMMUNE TO WEAPONS. If he had a bit of creativity he would've trained to be a karate master or something and just challenge every samurai he comes across. They literally can't touch him with a sword, so he could've beaten the crap out of them with his bare hands alone (or the bamboo sword) and earn a reputation as the great sage who doesn't even need swords to win battles. That was literally Yagyu Sekishusai's reputation in his old age.