You trained swordsmanship for however many years and you still can't even win a fistfight with a fisherman? AMATEUR
No seriously though, elf or not he should be fucking ripped with all these years of blacksmithing and sword-swinging experience. Even if he doesn't look it he should be physically strong too.