I will say, it is worth keeping in mind that Mori-chin isn't the one fetishizing it. But whether or not its a fetish doesn't really matter, in the long run.
Crossdressing, psychologically, is a very interesting thing. Some guys who do it don't get turned on by it, and some do. But regardless of how they react to it it fits in a weird third realm of being something a guy enjoys doing but doesn't really view as a fetish-y thing, regardless of if he actually gets turned on by it or not, because crossdressing is always more than just the act of being turned on by wearing women's clothing - it's the ability to explore fashion and makeup, the ability to understand women on a level beyond the surface, and an ability to express cuteness, something that for guys is almost never accepted.
Yes, it is unfortunate that Mori processed it as a fetish thing, but since crossdressing exists in kind of a limbo between fetish and non-fetish it's understood that's how a lot of people will view it. I personally have come to terms with it myself.