The phrase "I didn't do anything" doesn't make sense to begin with. For example, when someone says "I didn't do anything and my computer broke," from their perspective it means "I didn't hit the computer with a hammer" or "I didn't pour milk into the computer." They're only thinking of extreme acts of destruction, completely ignoring all the various things they did leading up to that point.