Just a conjecture but maybe because he's been living by his lonesome during that time, he never really treated the new world as something concrete and still associates it like an isekai world with the manga and otaku stuff he was hooked on back in his previous world (making bandits as less human and more like event flags). But now that he was able to associate with the Duke heir apparent's family, he started gaining a bit of humanity and started seeing people less as characters and more as living, breathing individuals.