Uh, that fight end was lackluster, even more after that rabid old man ranted about wanting to go to the end.
I don't think he dropped the fight because of the association, but simply realized that if the dungeon cracks the consequences on the outside too would be too much, he probably thought MC wouldn't have neded all that power.
Really annoyed from redhead attitude of continuing to think MC would abandon her for the father sins, but how I said it before is for a deeply rooted belief in Asia that the close family of a criminal must be dirty too, so she thinks MC would be the same, even more for him disliking criminals, not accounting how he only hates criminals and as her foster father is unlikely he didn't knew already and simply never brought it up for making stay clear of mind.