@wwasted of course I recommend the LN. But you have to start reading from chapter 20 of the first volumen when Masaru buys Satie since thats when the story began to drift apart.
Except for the Nera arc the manga has respected the soul of the source material and even if its different from the novel its not inferior quality. The tanned loli flashback from this chapter for example is filler since it doesn't appear on the source material but it was really good and well thought.
So feel lucky, you will get a new well written story if you read the LN.