Not trying to be sarcastic, but how can you be exiled to your home town?
Other than that I agree with what you've said.
tl;dr Not sure if there's a language barrier here or if the concept is somehow unique to the U.S., but your "hometown" isn't necessarily where you live right now, it's where you grew up. Haruka was living in Tokyo in Kazuma's place, so it makes sense to be "exiled" back to her hometown i.e. where her mom lives.
Longer:
If people grow up and move away from their parents' house, the place where your parents' house was is your "hometown". There's generally an understanding that people are comfortable in their hometowns as they're familiar with everything in it, there are chances that their former teachers are still at the same school, and that a lot of things can be explained about someone based on their hometown (i.e. you get a lot of your accent from your parents, but also a lot of it from the people you grew up with in your hometown).
https://www.dictionary.com/browse/hometown
If someone asks you "where do you live?" vs "where did you grow up?", they might get the same answer if you've never moved around, but the answer to the second one is usually your hometown.
The exact understanding of what someone's hometown is gets a bit murky if your family moved around a lot when you were a kid, or if there is literally no one of your family left in your "hometown", or if you moved away a really long time ago, or if you really hated the place for some reason, or if your parents were divorced with shared custody so you regularly swapped houses... So the answer is sometimes based on vibes, and sometimes you have no hometown, and sometimes you might have multiple hometowns...
But as long as you have a concept of a hometown, it makes sense to get exiled there. It's retreating from wherever you are and moving back in with your parent(s) / moving back to some place that's more comfortable for you.
So in this story, Haruka is currently living in Tokyo in Kazuma's place. Haruka's hometown would be where her mom lives / where Haruka and Kazuma first met each other as kids and were growing up there (until Haruka's illness happened).