The servants should be honest, and they are, but at least give the dude a chance. He just lost his memories, he's attracted to his wife, and is showing genuine remorse for his past actions. And from what I can tell, the wife looks to be attracted to him, far more than a sense of duty. If she really didn't care if he loved her or not when they married, then why did she almost cry in the first chapter?