If one ignores the Mary Sue MC, it makes sense that the guard and the butler escorted him.
For two reasons: 1) he is more likely to have conflicting interests, as he is an independent noble and he is no longer affiliated with that family. 2) If he has some grievances with his brother, or does something to the house, they have to sort it like two separate noble houses, and the owner of the house can't just order him around or punish him like a subordinate (and a younger brother would normally be subordinate to the head of the house). So for security reasons it makes sense that he is monitored until the master of the house says he can go around alone.