Instead of being so roundabout, if he really wanted to refuse (which would end the entire manga instantly because there would be no point then) he could literally reject them harshly once and threaten to work against any of them if they ever approached him again.