I would tell him, "My job description does not include partaking in conversations of a personal nature with clients. I suggest if you have a problem with your rent/girlfriend (whatever) that you discuss with those involved."
Really, he only does this because you don't stop him. SO, STOP HIM!!!!
