Really? Makes perfect sense to me. Her words implied the former heroes, as time went on, got so paranoid that she was stronger than them that they never let go of their holy swords at all; meaning no matter what they did, they had it in hand, not just nearby.
So when they ate a meal, they had the sword in one hand (probably using it like an oversized dinner knife). When they took a shit, they had one hand gripping their pants and one clutching the sword. When they took a bath, it was in hand. When they slept, they probably clutched it like a teddy bear.
They were so terrified of being caught without the power of the holy sword that they refused to part with it for even a second like a child's security blanket, because without it, they'd have pissed themselves in fear and run for the hills.
If that's not cowardice, I don't know what is.
I get where you're coming from but think of it in context of who she is and who they were. They actually had to fight her and her army of minions and she was known for coming back from the dead to start trouble over hundreds of years. Not to mention The Holy Sword seems to only exist because she was such a problem.
The manga makes her out to be a sad, cute dragon girl but we have to remember that before her pride was broken, she was probably incredibly vicious in the past. She was, after all, the Demon King. Families of victims of who she and her army slaughtered would mourning for years to come, legends and stories would've been told about her through generations and the way they talk about her makes it seem like those legends are pretty grim. She's not an innocent person just because she's sad now, she still has a rap sheet spanning hundreds of years and a body count higher than most Shonen villains.
If you tried to tell me that such a person decided to just give up out of the blue and not do ANYTHING at all, I'd probably never believe it myself. I'd be absolutely certain it was a trap of some kind. The only reason Mirio is capable of having any sort of relationship with her at all is because he met her under peaceful circumstances and while she was in a pitiful state. He even remembers that he didn't fight a single monster to get to her except regular forest creatures, and again even he assumed it was a trick when he first met her.
I don't think there's anything cowardly about being suspicious of such circumstances. It feels like common sense that you'd keep your guard up around a semi-immortal dragon that not only tried to kill you but keeps bringing itself back to life to trying to destroy the world.
All that being said, I don't doubt the story will vindicate her in some regard of her actions but given what limited information we know....they should absolutely be expected to keep that sword on them.