@Acolytus First, he's not saying that to himself, but to Fitts, whom have a worrying look at him, especially considering his early suggestion.
He's portrayed as being heroic in the eyes of his two companions since, no matter how you look at it, he still saved the child and able to at least converse with her. Nobody else would've been able to do that. That's a fact. They can't even speak the language.
He's not trying to justify himself nor is he saying that his situation is just as worse as hers either. Nobody hates his previous life more than he does himself. This is one of the reasons why he hates Hitogami at first (aside from him being shady as hell) , because he's always reminded of his former self he wants to cast aside. In this scene though, he's simply acknowledging that if he lacked the will to live back then, he should just end it by suicide. He couldn't, not anymore, so maybe he'll give her a chance to do it. "If living is too painful, I can just let it end here."
While what he does afterwards are him being a POS, he didn't become a recluse due to selfish reason, you know? He's been publicly humiliated.