Well in the Manga its not clearly said or visible, but IIRC in LN it is hinted that grandpa acted like that but not too over to the point its disgusting unlike in later arc shown that ridiculousness of the nobles(not the grandpa. Different person). Also its not like Grandpa act like that but...