If he's held constantly at home and that's not likely to change, you are better off letting him go now. Sometimes they come around but I know it was very hard to manage when the other infant was my own little one. Talk to mom and dad and see if they are willing to work with you. One tidbit of advice I give my new parents of infants is to not rush to pick them up if they are in the middle of something. For example, if mom is folding laundry and has three items left when baby starts to fuss, pick a song and sing while finishing up and then get baby immediately after... use the same song every time. Then you can sing the same song during daycare and it can keep the little guy from ramping up the cries while you feed your baby or change her diaper.
Don't hesitate to term if it doesn't improve though. This time is too precious with your little one to spend it stressed out.