Correct me if I'm wrong but I remember Seto explaining it along the lines of "if you are aware of them, and they know you are aware of them, it kinds of open a bridge between the planes and allows them to actively interact with the living, which never goes well because they still abide by the rules of the ghosts. So they can mess with you in ways you can't even begin to understand but good luck doing anything back to them."
That's for people who can barely perceive them. Now imagine what could happen if it's with Miko, someone who can legit see them.