@Weoooo イモ/芋 has a fairly long history as a pejorative modifier, and isn't inherently gendered. For example, rural samurai were mocked by more metropolitan courtiers as "imozamurai". Being rural carries the connotation of being poor, uneducated, and uncultured, in addition to being unattractive and out of touch with fashion. One can say imo-otoko to mean much the same thing as imo-onna.
At any rate, look at Iruka's face on page 18. He's literally snarling; his expression is pure hostility. He's not trying to "neg" her or even insult her per se, he just wants Koharu to stop wasting time on some farm girl nobody, and is trying to scare Sumire off to achieve that end.
It may be worth noting that he calls her variants of ガキ (kid/brat) in subsequent chapters (imogaki, kusogaki, etc.), which makes butterface kinda problematic unless we assume he's a lolicon.