Thanks for the link, but unfortunately that's not it! It's pretty close, but it's not the one I have in mind.
I think the isekai'd person was originally from Tokyo, rather than another fantasy world. I think he was an orphan or his parents died young so he was mistreated by his remaining family...