Ahhhh this one's relatable. I found out later that's called dyscalculia, and its why I had to re-learn a lot of base math in high school. There's work arounds tho- like keeping track on the knuckles on your hands to track things, or an abacus.
But until then.... well. When you're little, people forget you can hear them.