Those are NOT opposite statements, the top statement is obviously referring to intelligence while the bottom is obviously referring to strength
Oh look, 2 people thinking they are oh so smart.
The top statement says he doesn't know them in this world.
The bottom statement says he knows them.
You can say he's seen their strength with that warning rock, but believing that to be the limit for a race capable of building that town is just a stupid joke.
If author is thinking like you and the following chapters have them using rock throw, punching and sharp sticks as weapons that would be too outrageous for even isekai trash manga standards.