They were thought to be paid £10 million for the 2018 series, and are stepping down due to other commitments.
In a post on both of their Instagram accounts, they told fans: “A quick update from Team Williams… it’s been a very busy year so far and Vegas has been incredible - we cannot wait for June and July to do it all again.
“We’ve got some very big announcements coming soon - but we can’t reveal all yet. Sadly as it’s impossible to do everything, we’re gutted to tell you we won’t be able to return to The X Factor this year.
“We will, however, be working on projects with Simon going forward - and as huge fans of the show, we will still be tuning into The X Factor, voting and winding Simon up! AWxx”
Robbie, 45, revealed he was working on numerous other projects, including performing at The Wynn Las Vegas in June and July.
He added : "A quick update from Team Williams… it’s been a very busy year so far. Vegas has been incredible and we cannot wait for June and July to do it all again. I’ve got the massive @BSTHydePark show on July 14 and I’ve written the lyrics and music for the #RSCBoyInDress show with @guyachambers - opening on November 8.
"As well as all this we’ve got some very big announcements coming soon - but we can’t reveal all yet. (sic)"
The couple, who are parents to Theodora, Charlton and Colette, joined the show last year after judge Louis Walsh left after 13 years and they were joined on the panel by Louis Tomlinson.