Workers, because it's really not just one. (It's both stated AND shown he aint alone.)
And it aint implied that the building is complete, so the question about interior done or not is warranted.
Why would anyone assume the building is complete with interior and all just because the party came and talked to the mc while mc+other is working outside infront of the building,
and partymembers thinking about not remembering there being a structure behind mc+other? Nothing else mentions or shows anything about the progress.
Why do morons constantly start arguing without even checking the chapter first?
Brief check and you wouldn't embarrass yourself with stupid false statements.
Anyone with atleast a few years of experience in carpentry can point out all kinds of weird things in this manga,
but as previously said the timeframe aint one of those weird things unless the entire interior is done too.
(Because the exterior of such a small wooden building of that type involves no precision work and can be done crazy fast.)