It really isn't that hard to keep a castle. So what if they charge? If a castle is built well(and it should be considering it's the seat of a contested territory) and there's sufficient supplies, people can and did wait it out for years (average length is eight months, longest seige in history was 22 years). Walls were commonly 10ish ft thick and 40 ft high. They were built to withstand blunt force through specific architecture and were often surrounded by pits, moats, and other sorts of traps that make appraching a castle in any real number a nightmare. They're built to repel battering rams, trebuchets, ladders(which would be set on fire) and more! There should be holes for people to shoot arrows out of, pots of boiling water and hot sand to pour over the side of the wall, etc. There doesn't even need to be that many people to defend the castle(Even just a skeleton crew of 30 could defend themselves against thousands).
There is absolutely no need for anyone to leave the safety of the castle and go out on the field to fight on "equal terms" as if they'd no brains in their skulls. Castles are built to give defenders an advantage - why would anyone give that up?
If it's this easy to assault a castle, why spend the time and money to build them at all? How has the castle not fallen before now if it's so easy to attack when a single person is not around? Did he just never leave the castle? They're acting like their manor is being attacked instead of their heavily fortified castle. This 'battle' is just so unnecessary.