The empire's failure was to neglect the protection of the stone. If they didn't want to bother with it, they should have found a way to relocate it elsewhere. An outsider stealing it wouldn't necessarily care one bit how it affects the empire, which is exactly what happened.