"It was scary. I'm not a great swimmer anyway, not that we were swimming ... but they built a really impressive network of caves," the actor said.