His reluctance is from concern for his country's reputation in the world.
It's not that he's afraid of getting involved with a woman, but that he realizes that his actions reflect upon his society.
There is a difference between being an honorable man and being a beta.
Now, if future development are of the "Oh, a desirable female has become interested in me, ack, what shall I do?!?", one might be forced to reevaluate, depending upon his reasoning at that time.
One does need to take into account that he is successful in the eyes of the world; he was a salaryman, the only reason he isn't still is that he inherited farmland and decided to farm it.
As this appears to be where he was raised, he's aware of what is involved in this activity.
He's old enough to have a clue.