“There’s been trouble in the marriage over his decision to un-retire," a source previously told Page Six, adding the model has been the one to remain with the kids.