"We sat down and shook hands and the first line out of his mouth was, 'You know, my mom kind of fancied you,'" the "Yellowstone" alum recalled.