"You just want to shake her and go, 'Honey, they were never yours,'" the "Mad Men" alum said on Sirius XM's "The Howard Stern Show."