Sure they can... by jumping to conclusions when they only hear a minute portion of a conversation and conclude the worst case scenario for no actual rational reasonings😅. Has this family ever shown ANY indication that they're gonna get rid of him becuae of this? Nope. He even said that several times; once, even after the fact. So, him putting himself into this weird depression over it is pretty nonsensical (bcuz, again, even the mc knows that his family isn't thinking of such a thing). This more came off as "boo-hoo, I can't use magic. Im a loser" than anything else. This was just weird and nonsensical, bcuz his behavior here was completely irrational with literally no foundation to actually think that way.