Wait I'm starting to think that Shiraishi was a pirate (probably hanging around with Boutarou) before he went prison-hopping. It could make sense because, even though Boutarou was a tattooed convict alongside Shiraishi, there were a few chapters showing Shiraishi's culinary knowledge concerning seafood and his knowledge of the rivers in Hokkaido. That could also just be from clearing debris from the rivers but who knows.
Also, the "underling" thing that Boutarou mentions seems like it could connect to Shiraishi previously being an underling for Boutarou.