While the rest of the world was buzzing with excitement over the engagement of Prince Charles and Lady Diana Spencer, the couple themselves weren’t exactly as excited as they should’ve been. We all know the story of Charles and Diana’s marriage and divorce at this point, but many royal fans don’t realize that there was one family member who actually tried to convince Lady Di that marrying a royal was the wrong choice.

Her maternal grandmother, Ruth Roche, the Baroness Fermoy, was actually a close friend of the Royal Family—yet she still thought it was a mistake for Diana to marry Prince Charles. The baroness was a friend of the Queen Mother and also served as a woman of the bedchamber for the royal. When it came to rumors of her working with the Queen Mother to set up Diana and Charles, she told the Associated Press in 1993, “You can say that if you like, but it simply wouldn’t be true.”

In a new episode of the Daily Mail’s podcast, “Queens, Kings and Dastardly Things,” historian Kate Williams and royal biographer Robert Hardman discussed how Lady Diana’s grandmother was “the only person” who thought the wedding was a mistake.

“Diana told her sisters that the marriage shouldn’t go ahead and they said: ‘Bad luck Dutch, your face is on the tea towels. You can’t go back,” Williams said.

The historian remarked that Diana, who was only 19 at the time, later said “that she felt like a lamb to the slaughter.”

Baroness Fermoy, however, thought the union was a mistake. “One of the dissenting voices was actually her grandmother, Lady Fermoy,” Williams shared. “She said to Diana, ‘I don’t think their sense of humor or lifestyle is going to suit you.'”

However, the rest of the family—and the world—was “swept up in the idea” of the royal engagement, the podcaster noted. And, as Hardman shared, the royals themselves loved Lady Di. “By the time she leaves Balmoral, everyone’s going, ‘Well, you know that girl, she could be the one.”