That's what I figure too. During his mother's imprisonment it seems that he hasn't been able to meet with Shige and convey his emotions (regret/remorse/sympathy) and understand his POV. With that doubt of not knowing Shige's feelings plus the sense of responsibility and guilt that Sei may feel...