Reading back I finally understand what Ken said in page 40. Damn the translation was bad.
It was "You should've used your fists to protect that girl, dumbass".
Also kinda funny how in a later chapter (107), Ken reveals he knows Ban Phuong's backstory, and Ban Phuong gasps in surprise. Like, bruh, didn't you just tell him and his gang your story in this chapter?