@Baysiri there have been occasions where the author has shown that harmful ghosts can harm you.
For example, in chapter 15, when mc was in the train, a ghost with an axe was going around hitting everybody with the axe, and it is heavily hinted that it was hunting ghosts possessing humans.
The biggest clue that ghosts can harm people is that, the lady next to her was "not looking so good" and it turned out she had a ghost in her (unless it was her soul or smth, which would mean that what happened is even worst lol)
So they don't just pass through you, there is some danger, the series just haven't clarified how deep yet