Guys, from the very first chapter you can already tell that this story is kinda dumb, just turn off your brain if you want to read this type of manga.
On another note, he's a sword saint, not a fist saint, so of course he can't do anything without a sword even if he have trained and should have...