I agree that the manga is an edge fiesta but still my point is that he didn't start shits. Even if he could escape way earlier but didn't do it because then he would need to face 3 heroes on his ass. So that's why he wait for the right moment to strike back at those that wronged him.