@Argolok seems like the oni maids almost owns all the caring for the babies and he is the only one that can use the farming tool so he has little time for child caring, also in those little interactions we can see he is a doting father so its not like he doesnt want to interact with his children...