According to an Australian publication, James would find it "incredibly awkward to star opposite his ex-fiancee Jessica Marais on the show where they fell in love", after they called it quits in 2015.
The source told the publication: "They might be on good terms for the sake of their daughter Scout, but to be castmates again would be too weird."
The pair were together for six years, and had their daughter Scout in May 2012, before splitting in 2015.
James is now married to his Home and Away co-star Sarah Roberts, while Jessica has since moved on with beau Jake Holly, who she's been dating since 2018.
Packed to the Rafters ran for six years until July 2013, and the new season will pick up where the old one left off.
The reboot is set to feature original cast members, including Jessica Marais (Rachel), Rebecca Gibney (Julie), Erik Thomson (Dave), Hugh Sheridan (Ben), Angus McLaren (Nathan), Michael Caton (Ted Taylor) and George Houvardas (Nick Karandonis).
The beloved show is being revamped by Amazon Prime Video for 2020.
There have been a few changes in the family since the show last aired in 2013.
"Dave and Julie have created a new life in the country with the youngest daughter Ruby," the Amazon synopsis states.
"The older Rafter children face new challenges and Grandpa Ted struggles to find his place.
"As Dave enjoys his newfound freedom, Julie must reconcile her responsibilities to the family.
"They might not be under the same roof, but the same honesty, love and laughter bonds them more tightly now than ever before."
