I'm confused on the "our side" thing too. It seems to be what's actually said though. I took a look at the raw and it says the same thing.
あと 一年も雑用係をこなせば, 新しい人間を入れて, おまえも俺ら側に招待してやるさ
ato ichinen mo zatsuyou-kagari wo konaseba, atarashii ningen wo irete, omae mo orera-gawa ni shoutai-shiteyaru sa
After another year of dealing with being in charge of chores, (we'll) put in someone new, and also invite you to our side
あーあ!, もったいないなあ! あと少しでこっち側だったのによー
a-a!, mottai nai naa! ato sukoshi de kocchi-gawa datta no ni yo-!
Aah!, what a waste! Just a bit more and (you) would have been on this side
my guess is that by "this/our side", the forgers mean something akin to nobility or noble status, and they're baiting him with being recognized as a "true court blacksmith" (like them) only if he continues to do chores. It's not explained at all tbh.