"All I knew was at that time when Andy asked me, I just had no desire to be back on that network," the "Real Housewives of Potomac" alum claimed.