Prince Harry revealed he and Meghan Markle don't want to have a big family

The royal brushed off a suggestion that he should follow in the footsteps of a well-wisher in Ireland — and have five children. 

As the couple greeted fans in Dublin, Elaine Adam-Stewart, 43, told Harry, “My husband also has red hair and he gave me five children — when are you and Meghan going to get going?” 

Harry saw the funny side, Adam-Stewart tells PEOPLE. “He laughed and said ‘five children? — too many.’ ”

Elaine’s husband, Simon Stewart, who runs an IT consultancy, was not in the crowd outside the EPIC museum of Irish Emigration, but the English-born homemaker and her five kids —  Nathan, 10, Adelaide, 8, Malachy, 7, Siwan, 5, and Portia, 22 months, were — along with an England flag as they back the team for today’s big World Cup match.

Earlier in the day, they shared adorably candid moments with a couple of toddlers at Croke Park. Both Meghan and Harry share a love of children — and they’ve spoken out about their desire to start a family.

“Of course,” Harry said when asked about baby plans in a November BBC interview that took place just after the pair got engaged. “You know, one step at a time. Hopefully we’ll start a family in the near future.”

Last year, Harry said he was looking forward to fatherhood.  “I would love to have kids,” he told The Telegraph.

In a 2016 interview with Canada’s Best Health magazine, Meghan was asked about her bucket list items and said, “I want to travel more and I can’t wait to start a family, but in due time.”

This article originally appeared on PEOPLE.