Dowun is a very goal oriented, success driven, efficient kind of guy. He's the type to purposely avoid building more connections or feelings than is necessary to get the job done. I feel like what we're watching is Dowun gradually getting an inkling of affection for the empress despite himself. Despite the fact he's always able to give a cold logic reason for his actions, I think at times there's a little more to it.
On the flip side, we've seen the empress build immense affection for him when they were young, but now that they're older it's started to wane with doubt and skepticism. Moreover, her life savior has now reentered her life. However, I get the feeling she can't truly be indifferent to Dowun even though she may wish to. At times what he does pushes her away, at other times it draws her closer but regardless his presence moves her one way or another. Makes sense, as in his own way he also saved her life. While he wasn't selfless and she's had to make sacrifices, it's clear that if he hadn't entered her life she would have been forgotten and wasted away at the bottom.
Its quite interesting to watch these dynamics play at the same time. Right now I'm reading it more as a drama than a romance which is fine for me. In this particular scenario I think a slow build seems more fitting.