I might have read the Villainess reverses the Hourglass a bit too much; but the fact that the only thing he always loses is his watch and he knows the exact number of times he lost it, plus the words he says in the early chapters, something along the lines of "I'll protect you this time", could be a hint that he time-travels using his watch, and this has happened 6 times already.