Be true to yourself and say yes when he asks to spend time with you. Otherwise you will regret it. The hurt he may cause you during that time together can never be as bad as the regret from saying no.