There's no way a duke's daughter would marry a gardener's son. Maybe if the MC dude used his knowledge from his original world to make himself a man with wealth beyond measure, it would work. Like he would first revolutionise the steel industry and then the paper industry. Under such circumstances, the king might make him a baron easily and the duke would be happy to get a share of the profit through a marriage alliance. However, he's happy being a gardener, so I don't expect him to revolutionise anything at all. Not to mention by the time he had actually pulled off the industrial revolution, Lydia would already be married off to someone else. Technically even the maid would be too much for him, being the daughter of a baronette, but it could be at least somewhat imaginable.