I like this. It's very different from the regular isekai. He's truly feeling grief over his lost years and experiencing nostalgia, without letting it get to him too much. Let's hope it stays like this, but it's looking very promising.
As for the actual isekai-ness;
A) No, it's not isekai, because he didn't go to another, unknown world. He time travelled.
B) Yes, it's isekai. While he may have time travelled, the premise is them fighting in the 'Shadow World', which can be counted as a different world.
A vs. B, FIGHT!