Agh! Run and run fast!
I am not super confrontational either, so I would probably use email or text and let them know "Due to changes in my schedule, I will no longer be able to provide care. Sorry for the inconvenienece and best of luck in your child care search."
If they have stuff at your house you can offer to mail it or have them pick it up when you know you won't be home and just have it ready to go on your door step or something. Hopefully, you won;t have to see them and you can move on and get some sleep!
