I read ahead, and it actually does improve noticeably after this incident so just trust us and wait a bit.
@ethian78 is right that it's uphill from here. If I really had to quibble about what I've read so far it's that...
Personally, I think that at Lyle's age the First was kinda an outright loser compared to Lyle just being a pathetic kid, and he's only shitting on Lyle now after he had a lifetime to shape up.
Also the harem is outwardly born from a really shitty gag, but that kind of thing feels par for the course for this author.