I appreciate that our protagonist usually doesnt bother correcting people who make assumptions about his money and living situation. He doesn't care what people think. But this punk is making a nuisance about him, so he correct him. And then says, with a very casual tone- "I don't really care what you're doing so long as it isn't bothering me, because thats not my job."