Retiring to a small countryside village and living your life in peace... CAN NOT BE THAT DIFFICULT.
Sure... monsters, bandits, and assholes are aplenty... and they stay aplenty even without magic.
Just... go live your life FFS... stop making your pain everyone else's problem... find a little circle to share your pain and help each other, but... STOP THINKING YOU SUFFERING ALLOWS YOU TO DO THE SAME TO OTHERS.