@Rurichou
I get that the combatants here are non-human, but the people who live in the castle know this and should've taken non-human abilities into account during construction. Plus, it is canon in this world that there is permanent defensive magic on the table. Why not use that?
The castle must have been built for some reason beyond comfort. There doesn't seem to be enemy humans around and yet they still use that castle as a defensive fortification in a contested area. The castle has been attacked by way worse than a couple of "rock golems" (note the ant army), so they SHOULD be able to handle them easily enough. This world is based of historical European society and architecture. If the enviornment of the world changes(ie the addition of monsters and magic), the people should be able to adapt. It makes no logical sense for someone to build a useless, expensive castle in the middle of a warzone as a military seat if it is useless in the face of any real attack, and how they would've supposedly survived all these years to leave an illustrious legacy makes even less sense.