I have too say, this could've been a lot better if it wasn't:
1. A harem. Sometimes a harem can add to a story, especially if its a part of the culture of the country/place inside the story. However, I can't really see any contributions to this. It would've been interesting to see how he regained his relationship with his lover from his previous life.
2. Lacking a struggle situation. Sure every now and then they may struggle against someone, but nothing that REALLY sets them back. They'll overcome it relatively soon.
However, its not too bad.