I tried reading the novel to see if there was a reason for this, but there's nothing explicit. Reading a bit between the lines, I think what he meant was that since the imperial army couldn't conquer them, they were a much bigger threat than the empire. I don't know yet what "breaking through the heavens" entails, but my guess is that they want to ascend to the higher world by force, and in doing so might destroy or damage the world they are in, so if the Mountain Manor were still around, they'd be able to stop them.