The "Balthazar" owner — who recently claimed Corden was "abusive" to his staff — slammed the actor for dismissing the claims as being "silly."