I've always appreciated the fact that Baku was physically weak. It made him appeal to me more as a protagonist, since he was playing death games in a world of superhumans with a huge handicap. I feel like it tied very closely to the theme of the manga where if you can't enforce your winnings in a gamble, then it means nothing. He had to perfectly leverage Kakerou's immense power to actually enforce his winnings, which is important because Kakerou was very strict about their rules (Lacy "won" and got his ship to sail, but had no protection from Oofuna and Hachina). Even then, Baku was often put in compromising positions (Abandoned Building, Labyrinth, Tower Game, Portoporos Nightmare) and had to play perfectly to avoid superhumans (Rodem, Minowa, Billy, Hyogo). Even in games against people who we can assume had "average" phyiscal strength like Sadakuni and Lalo, Baku was at a disadvantage (as shown in the final arc of Air Poker where Lalo considered just having a stamina battle).
It's also nice that it's confirmed that it's a debilitating weakness and not him simply being a weak skinny guy as a gag. Ultimately, Baku's heart weakness WAS a crippling one, but he was able to avoid ever having that weakness exploited by his opponents. It makes his feats very impressive, and I think ties very nicely into a theme the manga has been hammering over and over. Same with the concept of a gambler that only wins doesn't exist; everyone loses sometime, and everyone ultimately will die. But considering how The Prince Bee goes I'm pretty confident Baku knew that ultimately Hal would come back stronger than ever, and he was going to be giving an even stronger Kakerou back to him.