"I had no choice but to punch back seriously, otherwise I would have died"
Bullshit. Your HP is peak S ranked. Your Strength is peak S ranked. His Strength is A ranked, maybe peak A rank. Even if you decided to not be "serious" and gave a halfhearted punch of, let's say, B-ranked strength, it is doubtful as to whether or not you'd have even broken a bone, let alone died.
The person who adapted the novel into a manga really knows how to add unnecessary lines huh.
On a side note, I'm a little sad that the manga decided to have Ryouta take the jealous route rather than the creeped-out-by-the-homo-sounding Neptune route. None of the jealous shit happened in the novel version, and rightfully so, since 1) he's living a very fulfilling life with good relations to multiple girls, 2) Neptune acts homo as all hell from the beginning to the end of the encounter, and only temporarily relieves Ryouta by kissing the two girls with him, and 3) Ryouta isn't really the jealous type anyways.