Remember that the account of the travellers is from those who hate them so it is unreliable. Who knows what really happen. They might just been used as scapegoats.
Technically I don't think anything the travellers have been said to do have necessarily been done out of any ill intent. The dungeons/monsters-thing is questionable, but there might have been a reason for it, have read other stuff where dungeons siphon off miasma or similar.
Mostly the problems seem to be unintended consequences from actions taken by people with extreme power. The reason for wanting the travellers dead seem to be to stop these unintended consequences, and potentially stop someone who would actually use their powers for ill.
Which seems reasonable as one travellers supposedly wiped out local fauna, from what could have been a wish to improve the food situation of the world. And the floating island was a seriously dangerous thing, even if the intention seems to have been good, or at least neutral. It would be pretty impossible to figure out any negative side-effects of your actions if you had that level of power, even if you wanted to make things better, and you would seem extremely scary to anyone who didn't have your level of power, especially since you are from a completely different world with different moral values.