The reason I would say it started as NTR instead of just Cheating is that Susa and Amane were manipulated into it while it was all planned by Saku and Miko. Always making sure just one more thing was needed so they're "equal" and "we're even"... Also remember that Miko wanted a specific act that Amane hadn't done on Susa to insure that it would injure Susa emotionally to see it performed by his "girlfriend" on another guy. Apparently Amane had never given a "cleanup blowjob" to Susa and he broke when he saw her doing it to Saku and Miko intentionally pointed out what it was.
After all of this happened (22 chapters into a 41 chapter manga) then, yes, it became standard swinging. (Except that as I understand it swinging is supposed to be purely physical instead of one pair of them actually loving the other pair.)