I mean, even if they aren't refuting it, they aren't actively pursuing it either.
It's a level of ambivalence about the matter rather than jealousy or other such feelings.
I wouldn't worry about it impacting the story in the long term.
unless this does end up being a "must do XYZ to leave"...
And here I had thought they just had no interest in Will romantically
They knew they stood no chance against adventure sure but they don't look opposed to being concubines.