The source added that Mary-Kate, who runs two highly successful fashion brands with twin sister Ashley, rarely took time off for a holiday.
"Olivier loves traveling to tropical places with his French friends," they revealed, adding, "He travelled without her several times."
Mary-Kate and Ashley started working as babies, even before their first birthday. The twins rose to fame on popular sitcom Full House and then went on to star in their own films before entering the fashion industry.
"[Working hard] comes quite naturally for us," Mary-Kate told Net-A-Porter's fashion magazine The Edit in 2017.
"We don't need so much time to sit and think and ponder."
Mary-Kate then requested an emergency order to file for divorce, due to fears that she was being forced out of her and Olivier's apartment, but her request was later denied by a judge.
Mary-Kate and Olivier married in 2015 in a private New York ceremony and Mary-Kate not only gained a husband, but two stepchildren too.
In 2017, the notoriously private fashion designer spoke to Net-A-Porter's The Edit about her marriage.
“I have a husband, two stepkids and a life; I have to go home and cook dinner. I ride horses on the weekends," she revealed.
“You find the thing that helps you relax and if you don’t have it, you have to look for it. Or you get burned out and then you’re not productive.”