I used to feel terribly guilty.
I always gave them a lot of notice and tried to group appointments all on one day or make the appt. for the end of the day so I only needed to close an hour early. When I felt really guilty was when I got sick and had to take time off with very short notice. It's hard! But it's very important to schedule self-care in along with everyone else in your household!!