I don't think the butler loves the lady as a woman. You don't blurt out being confessed by others to someone you like/love romantically.
I think the butler is the one who thought himself is the biggest hurdle of the lady getting the lover she deserves. FMC affection to him was assumed as she being pitiful to him while in reality it's the FMC who can't grow up without him.
The last outing without him is probably the way her parents assessing the lady, to see if her needy behavior is salvageable. Do note everyone in the mansion knew how demanding she is to the butler, and they couldn't do anything.
Besides the usual trope between butler and lady, I would like to see the butler actually fall in love with another woman, regardless of status or temperament. Falling in love is just that weird, and would spice things up for the butler to learn emotions