@NinjaGoemon: Even besides personal space, part of it is him knowing it's an illusion. He's not willing to force it off when it looks like Suzu, yet he's also too monogamous to be comfortable with what he knows is not Suzu.
@Stormtomte: Matsuri actually could have stopped it at any time, but chose to accept whatever Tanumaro was willing to deal out as punishment for. That detail does establish his feelings as just "very uncomfortable".