"Tory is scared for his life and safety in prison," a source told Page Six. "He feels like he is a direct target because of his celebrity status.”