It would depend on whether or not your contract said "during working hours, only" or you had contraced hours. YKWIM?
Here, I would not have asked. I have opening to closing hours. Fixed. So what they do when the child is here is none of my business as long as I have correct contact information.
It is hard to say if it was wrong without knowing the deal you had with the family.