Shizuka is wearing Shrine Maiden clothing here, so I'm assuming they're meant to be a woman. Shizuka is also typically a feminine name, though it is unisex.
I too always thought he was male, also because they call him a priest. However with fisherman's clothes it gives decidedly a feminine aura.
Maybe this is the work of a flow, or else they will make the gag "oh I thought you were a man"