Seems to me like the matriarch, great elder, 2nd elder, 3rd elder, and mc have all been aware this whole time and were faking their exhaustion (or will have mc make them recover with finger of nirvana) to bait the blissful temple.
I think this because most obviously when the mc met the 3rd elder to get them all to meet, but also because when the 3rd matriarch asked why the 3rd elder attacked the enforcer seriously, and when the 3rd elder saw the 4th elder was using the defence thingy and thought that the matriarch and great elder might actually start fighting each other seriously.
If this is true then I'll be satisfied but if the mc really was just clueless this whole time I'll be really disappointed