Well, probably what keeps us humans from rebelling against other humans; violence, lots of violence.
We don't know the actual details of the deal between humans and the ogres and we don't know who enforces the deal and how. If, say, governments are involved, rebelling may well spell the end of...