I don't know how much I trust this Ishmer. His chivalry seems to mostly extend to combat as he has no objection to stalking a girl that was both forcibly summoned and placed into an arranged marriage despite knowing it is not her custom or desire. I think that regardless of what he said, he will still disclose Sakurai's abilities as he see it as his noble duty to the country. Finally, if he knew the plan by the summoned to return to their world, he may try to use a slavery collar on Takanashi to prevent this, just as the king has on Watanabe.
Maybe I'm wrong, but with how infatuated he is I don't think he'd let go of her easily.