It's likely that while his software (mind/memories/personality) might be an adult, it's now running on a 3 year old's hardware (brain/hormones/chemicals), so he's no longer as capable/mature as in his prior life. Or so I hope the story goes. The age regression/reincarnation stories where the young brain affects the MC's new mentality (and the MC having to come to terms with that) to be better/more compelling than the stories where the MC is basically just a tiny adult people mistake for a child.
What was even the point of showing his adult life and form in the previous world, if none of it matters and the MC doesn't care about the previous world? He moved on pretty damn quick. Might as well have just started from when he was in our world and not bothered with his previous life at all.