Second chapter has expanded the premise nicely. Many people are experiencing delusions of returning from an isekai, enough to get a medical diagnosis. There are two possibilities- either they're mass delusions created by stressful situations in the real world. This would explain why those suffering have differing accounts of their time in the isekai, or varying levels of detail.
Possibility two is that a bunch of people were genuinely brought back, without their otherworld skills, and they have ptsd. So they've banded together to create a workplace/support group to make the best of their situation. They might not even have come from the same other world.
Or there's a mix of options one and two- some are mentally ill, some are the genuine article. The tension between the two options is a realistic setting for a world with returnees. Most folks would not believe you, and it would drive you a little crazy.
Premise aside, I don't know to what extent belief/disbelief is the primary plot point. Maybe as he gets closer to his coworkers, they slowly accept they're all genuine returnees. This also seems like a harem template, so I bet a lot of the "plot" is just lucky pervert moments and the work environment becoming like Fleetwood Mac recording Rumours. It's a perk when there's another theme beyond fanservice!