Charles had the Queen's trusted confidante, Angela Kelly, moved from a "grace-and-favour" home because he reportedly "didn't want to live" near her.