The wannabe yakuza agent totally wants to control many aspects of Narumi's life. She ought to be thinking that Narumi's career can only succeed by following her instructions to the letter. For an agent such a mindset would be essential (because an agent also need to think of their own career), but only to a certain degree. Taken even a little bit too far, it's putting the cart before the horse. Because the agent is the servant, the actor is the master. Where Narumi lives, within the same city, is none of the agent's business. Narumi shouldn't need to give any reasons or excuses and the agent shouldn't say a single word against it. The agent should just stick to the professional side of Narumi's life, and that's it. The private side only plays a role if she's in scandals or neglects her health, for example.
Too bad Narumi is rather timid, even if she's always working hard and is proactive. The agent wants to be yakuza, so of course she'd exploit any hesitation from Narumi and push through like a bulldozer with her opinions.