So it confirms another theory I got in my head some time after my comment : the Atelier is his home and the girls and Olugio are his "family" (not into yaoi anymore, but demn Olugio and Qifrey more and more look like two dads and their daughters, everything is telling they're two face of the same coin, from the color scheme black / gray-white, to the exchange of their hats, the fire-water opposition, how Olugio looks grumpy and a loner but using a heartwarming fire-magic / Qifrey being friendly and happy-go-lucky but able to cast powerful and terrifying water spells...)
Saving (and being saved by) Coco made him truly understand how precious everyone is to him, and thus how he doesn't want them to be involved with the Brimhats, way more clear, and coherent.
But doing so, he's currently carrying a dangerous burden. That explains Bel-Bel's line "beware to not destroy what you're calling your home" ; as he used a forbidden spell on his own friend, will we see him using it on his students? Plus, it explains the page after the memory-wipe, when we only see his dark silhouette.
Ah, now everything is clear, I like this duo even more. Qifrey's going on a dangerous route, he's like his color scheme : gray. Still a nice guy who wants to protect his family, make his student improve and just, live his life. But, still heavily scarred by his past, and ready to use forbidden spell to either wipe suspicious people (Gramps) or protect his own family (Olugio) or, worst, leave his students alone to pursue an enemy (Romonon episode). Definitely, Olugio and Qifrey are my favorites characters.
Ah, now I'm curious to see where all of this is going! Really, I like this Manga. Characters are well written, and if I'm still curious about Tetia's past (as they all have in common the fact they're "rejected" from witch society), and want to see more about various magical symbols, it doesn't bother that much to change my mind about the the story.