Except that here, his whole family is telling him straight to go away and die. They confirmed that it isn't a joke.
Then they were not jokes when they tried to murder him time and time again, were they? They've been just as serious all the other times, but for the sake of the series it can't happen. They do not love him. But the whole point of the manga is to say: look at these insane people that could not possibly exist in reality, isn't it funny?
In this chapter when they say it isn't a joke, the actual joke is in that him leaving somehow gets them so much money that they can get diamonds and gamble.