"The Master Swordsman is followed by the King of Illusion" Well, a phantasma tamer is definitely a "King of Illusion"
Also I'm pretty sure he isn't aware of the other phantasms that were in his previous kingdom (such as Gaia) and wasn't aware that by leaving, all the benefits would disappear with him. He has tamed those three specifically and it was only them who communicated with the others. He didn't seem to acknowledge them. He may slowly tame the others over time when encountering them.