This story is so bad, whenever a Japanese novel author steps into isekai it seems they involuntarily turn the whole story into the same copy-paste stuff.
Guy buys female slave, has only one special skill but then somehow is able to learn everything else through that one skill(or because of Deus Ex Machina), becomes a badass adventurer and then is involuntarily involved in politics (the adventurer step can be skipped if the guy is either a. a noble from the start or b. a merchant instead), turning the complete story into a cluster of genres and stories, leaving everything half-baked and superficial + boring because the character is already so overpowered that he can't lose, or if he is losing, Deus Ex Machina comes to the rescue.
The only good stories that at least partially evade going down this path are the revenge stories, where the main character has to overcome his humanity to survive on his own. But even then, there is too much Deus Ex Machina that makes you want to skip through all of the story because you can predict how it's going to turn out and reading just wastes your time with empty pages covered with words and numbers.
It could've been a good story if they at least saved the female divorcee as his bride, and he would've had to build up the noble family with her help from the start (as both are pretty much not accustomed to being a noble, it would've been a better story seeing them struggle with the sudden collapse of the kingdom and the power-&administratory vacuum).