"We've had a couple of toxic people, and because of my fear of confrontation, maybe I didn't deal with it quickly enough," the actress said.