The people who hate his family have no empathy or knowledge about mediaval society at all. You guys just don't understand the intense of abandonment. Wouldn't you want to keep your loved one close to you ar any cost, especially when they could have gone forever? Yoda already said that he had no family in Japan anymore. His responsibility is to his family in another world. The way he acted made his family feel insecure so it is partly his fault, too. He failed to make his family trust him.
I sympathize with his family a lot. I have seen many cases where the husbands abandoned his family, especially with immigrants. The husband moved to the US with his family, then he went back to visit his hometown alone, met another woman and decided to never return to US again, abandoned his family and sending the divorce paper through mail. The wife and children are heartbroken. That happened to a friend of mine and many others Men are easily distracted by new things and women.