It's understandable that he isn't ready. What happened to his isekai mother, still linger almost everywhere in his mind, knowing the past, he wasn't sure if he could protect the ones he care when the time is right.
From what I see, its not that he is being a wimp, but I may agree that he is somehow being a coward to that in a little bit. But that is probably that he is being caution towards what's to came.