@ZerbosG
This is my final advice (until my next final advice). The extent to which you follow it is up to you of course, but it's worth considering.
1. Treat him as politely as you would a stranger, but not a stranger you are actively attempting to get to know (as in, don't initiate dialogue).
2. Don't fall for his bait. Just like on the internet (or the other way around rather), people say and do rude/harmful things in order to elicit responses.
3. If a direct confrontation occurs, act as if you are mentally over the situation. Just saying things like "yeah, I don't really mind anymore" or "I realized it's not that important" do wonders in showing that you're not bothered by his present or past actions.
As with anything in life: if you don't let something bother you, it won't. And when other people are involved, the next best thing is to
act like it doesn't bother you, even if it really does.
That's not intended to sound unsympathetic, it's merely how it is. We've all had things that have bothered us to varying degrees. Your feelings are very real and I can empathize with them, but venting them outside of your bully's sight, as you're doing now, is better than venting them in front of him.