The cat didn't summon him, and the samurai chooses who they follow. The swordsman himself claims to be the original owner of the sword anyway. So this argument of master based on ownership of the sword holds no bearing. After all the point is moot as the cat likely acquired the sword as a spoil of war, so he is also a thief in a way.