Eh not quite. Pilgrimages were fairly routine and it wasn't uncommon for anyone to make one. While it's true that a middle class didn't exist, people could still sell their labor en route to make up the funds - and they did. It's not tourism as we know it today, but it was real. Something like tourism as we know it today did exist during the Roman era, especially between Rome and Greece. So it's not a purely modern phenomenon. You need only look to Pompeii, which was a tourist hot spot before it was buried under ash.