Oof, I am still hoping for some healing, but it still hasn't happened yet; right now we are still in the 'angst' phase.
Still, both the FL and ML are quite pitiable... FL was abused by her family and had an apathetic husband (but all of the servants love her, so at the very least she wasn't lonely, and she had trustworthy companions), while ML has to navigate the world with no memories, and is being constantly reminded that he was a horrible person in the past. That said, since his current personality is so kind, it's likely that this is his "original" personality, and it was the stuff he experienced in life that made him a terrible person.
IMO this story might have started off stronger if we saw this from the ML's perspective first, since he basically needs to start things from zero, like us as the audience. He gets thrown into a world of grandure and learns of the duties he had, and needs to start all of them from scratch. Then, as he interacts with his wife (FL), he learns how horrible of a person he used to be; thus we enter FL's perspective to see how she had experienced it. That would have been a more interesting development, and his "past terrible personality" would become an early twist that sets the main conflict into motion.
Still, it seems like the ML wasn't a bad person per se, just apathetic, and even now we are still picking up the pieces of what kind of person he used to be (which is probably a set-up to show that he had his reasons to become his apathetic self). But the fact that they chose to start from the FL' s perspective seemed to cement the impression that the ML was a terrible person, which is kinda unfair for him (especially when people are talking about dropping this already when we barely started). And, well, the ML's character arc is 1000x more interesting than the abused FL cliche, and it's pretty obvious he will be going through a dramatic character development and having to take the most active role, while the FL will remain a blank static character that just sits by and watch what he brings her. Just that alone makes it clear that the ML is the protagonist of this story, not the FL. Regardless, we are still at the early parts of the story, and I am looking forward to see if it might overturn my expectations.