Thanks for chapter!
In LN it explains more that the 'accident' was done on purpose. One of the things the 10 main families use to maintain their authority is the statement that "The 10 main families are superior in all magic." Meaning all family members have to be able to use and be above average in all magic even if specialize in one. That's why after Tatsuya beat a member of the 10 families in the school competition Juumonji and Saegusa were ordered to not only win Juumonji's match but do it in a flashy manner to show off the strength of a member in the 10 families. Due to a loss of about 40 family members from the family heir being kidnapped the main family looked to adopt people from branch families to increase their numbers. However, Tatsuya having all his magic ability taken up by Regrowth couldn't do other magic. So his aunt as the family head made a magic calculator spell as an experiment that should allow him to use above average magic with his mother being the one to apply it to him. However, she had to remove higher emotions to make enough space for it. Wanting to make sure it worked she removed every high emotion he had, with only sibling affection being left as a means to control him. Result was barely able to use other magic, loss of most of his higher emotions, and being treated as a failure and black stain by family and servants. Hence his ability to become poker faced and calm due to an emotion just being changed to data when it reaches a certain level. So while he might get irritated at someone being annoying, ticked at someone being rudely offensive, his emotion will shut down and change to data at furious. Only exception is if it ties to sibling affection. So when he caught guy sabotaging Miyuki's cad and had pulled him out and about to messily carve him up with the aura of an enraged demon king it was because Miyuki was endangered which tied the anger with sibling affection. So while he can like Haruka as a friend, if it became love the emotion would be converted to data making him seem not interested in her at all or as very cold.