@sjmcc13 He didn't learn swordsmanship, he unlocked skills. How much simpler or quicker is irrelevant, this body was originally a hunter. If MC can fight against monsters right after unlocking some combat skills, he should be able to utilize other still he had before. Not all skills have to be activated, some are passive. His buffs are passive as well. For instance, his jump was over-amplified by strengthening without him being aware or wishing for it.
MC was not in the town, but somewhere on the outskirts of it, right next to the forest. Other villagers avoid and hate him, he got yelled at right after entering the town. This should be a good enough reason to be on high-alert. Now thinking about it, he never questioned why the original body was dead in a place like that, so much for +5 intelligence.
Even though MC thinks he got ambushed, it seems to me he was just being retarded. Based on the flashback, these three dragged the original owner into that cave to kill him. They probably simply walked behind him without putting too much effort into masking their movements.