So the actual issue is that he's not using his sharp-tongue with the people that put him to work.
Just tell em that you'll quit and become a gardener if they don't let you marry your girl and they'll shut up.
@Nerdysylveon make sense since in the light novel it is only in the main character perspective. You didn't get Eddy side of the story until the very last chapter where he recaps his side of the story. The manga is completely doing the events out order that it is ruining the story telling. It to point it show a scene that doesnt take place until Book 2 and it after they get marry and he trying to protect her as his wife not as his fiance.