I'm pretty sure it has something to do with his spirit core or nucleus that happens to limit is strength, what most he have is experience and he only fought himself in that world not knowing other's fighting strategies and styles.
I don't feel like you're trying to be rude or anything, and I am definitely not trying to be rude or condescending when I ask this; I am asking you honestly: do you understand the concept of "A Billion Years," in the scale of time?
Over 1 Billion years ago the Earth was in the
Middle of the Proterozoic Period; 538 Million years ago, a little more than
Half of the billion year time frame, is when
multicellular life first occurred.
A billion years is enough time for him to have evolved out of being human, and enough time for him to have either hit his "peak," or to have compensated easily for the barrier on his soul.
Modern humans evolved from mathematics
not existing, to spaceflight being possible in less than half of 10 thousand years, and that's less than a single percentage of one million years... which is only 1% of a billion years.
Even if the TL messed up (multiple times) and only meant a million, it still would be impossible to fight any person in existence and have a
TIE.
Even if his spirit core had a block, he would either plateau at some point before then, or he would overcome every human in existence.
TL;DR:
Nah; billion is too long a time-frame.