I kinda feel that this "that man" is the MC's sensei (Vincenz), and probably the one who wears the black robe that met with the MC. If you compare Vincenz and this black-robed man, they both have similarities, like the hairstyle, earrings, and even the color of the hair. The only thing not connected to Vincenz is that he hates the irregulars, while this black-robed man is the one who is spreading/making irregulars (If he really is Vincenz, maybe he had some reason for it? Probably that's the only way to save the world from irregulars, like after he trains the hero (the MC), then he becomes the villain?). Remember, in the early chapter, he reminded the MC not to hesitate to kill him if they had a chance to fight. Vincenz was also the one who had this power of time magic, and he tried many ways to fix the timeline, but he always failed.