I agree. I like it when a series doesn't feel like it needs to adhere so strictly to a canon world or whatever. Spin off to different times, different worlds, whatever might make the story fun.
I also find it kind of telling that even in an Edo story, Kariya has to fall back to a rich, powerful man being the solution to Yamaoka's problems. It's something I've noticed with each volume, that Yamaoka can only do the things he does because he's cozy with the rich and powerful. It feels kind of unintentional too, since it goes against the themes Kariya presents through surface text.