Yeah to me this look like the story lifted from somewhere else, but done in a much more rushed manner. I understand the feeling/desire of authors of getting the story going, but to me it feels like I'm reading the Amusement Park version of a better story. There are quite a few stories out there about isekai-at-will peddlers, and they pretty much all did it better.
A good example of this better-not-think-about-it problem would be the merchant who bought the sugar. There's no way in hell that anyone would make such a deal with some unknown person for a luxury consumable item with a form/appearance that's different from the known measure. A good merchant would have exchanged the sugar for a deposit and rights of sale, with a promissory note for the earnings later. That dude outright cashed it out on the spot, and an amount that allowed to buy two high-value slaves.