Indeed he does have his priorities.
Although it seems that his perversion is more an outlet for rage, rather than actual interest. I mean, he actually got caught off-guard just by Fury blushing from liking his cooking. If anything, he's relationship starved in his past, probably.