Holy fucking shit Fujita it's not even chapter 30
Okay, so, after a small bit of research it seems the "I am your captive." meaning of peach flowers comes from Victorian flower language, where it was meant as "I am completely infatuated with you."
However, as we saw last chapter, Kotoyo, Ginbei, and Hyoudou attribute the meaning of the flowers to Ginbei and Hyoudou, though Kotoyo asks if they can all be "unrivalled" together. Furthermore, Kotoyo talks of how she wants to take care of everything so Ginbei can focus on his own journey and was scared he might abandon his journey for her. With that in mind, and especially looking at how Hyoudou reacted, the real devastation was that Ginbei assumed he was holding Kotoyo captive in a sense (hence his constant apologies), but in reality she felt as though she was holding him captive from his own journey in a sense.
In this same chapter Kotoyo also says she and Ginbei are twin stars, which seems to say she saw how they both shared feelings and both saw each other as holding the other captive.
Side note, but I also just really love Hyoudou's reaction to the whole reveal. before he can even process everything and is still crying into the book, he's putting on his suit jacket to go see Ginbei. Plus he leaves as soon as he gives Ginbei the book, not letting himself even see Ginbei's reaction. I think it really goes to show the strength of their friendship beyond just the petty rivalry it may seem like on the surface, because if that was the case witnessing Ginbei's lowest moment would be an opportunity he wouldn't miss.