Well, with Ieroh being called Yellow is just a quick jab at him being Asian, but for the symbolism, his whole being in America was because he was too afraid to follow his honor-bound comrades to the grave. And by fleeing from Japan with Murasaki, he essentially becomes a "yellow"-bellied deserter as he ran away from the responsibilities/code of his past life (as a samurai/retainer?). Also, aside from his first encounter with Blue, he hasn't once used his sword, preferring his rifle. You know, a weapon that can be fired from such a distance that he wouldn't really need to confront anyone up close, a coward's choice. But, yeah. That was in the beginning, so Yellow doesn't really have that 'umpf' as it once did, and it now serves just as a reminder of what he once was.