In his recent appearance on The Reluctant Traveler with Eugene Levy, Prince William opened up about his relationship with his grandmother, Queen Elizabeth. He admitted that he grew closer to the late Queen and Prince Philip in their older years, while relations were “quite formal” when he was young. Robert Jobson’s forthcoming book, The Windsor Legacy, offers more insight on the relationship between William and his grandmother—and according to palace sources, she grew “irritated” at one of his traits.

“Although she loved William and saw much to admire, she told insiders that she feared he might become a ‘celebrity monarch’ rather than a dedicated one,” Jobson wrote.

The Prince of Wales has been vocal about putting his three children and wife above everything else—a viewpoint that the late Queen, who viewed the monarchy as more important than anything, didn’t necessarily share. Jobson cites one example when Queen Elizabeth asked her grandson to fill in for her at an event just months before her September 2022 death.

“His rigidity, according to a royal source, occasionally frustrated her,” the veteran royal journalist wrote. Jobson continued that the late Queen was due to open a new hospice near Windsor in July 2022, but “she was feeling particularly frail” that day.

As Queen Elizabeth “had a particular connection” with the facility through one of its retiring staff members, “she didn’t want to let anyone down.” But when she asked William to fill in, “he cried off, citing fatherly duties.”

As he noted in his appearance on The Reluctant Traveler, Prince William and Princess Kate try to plan royal events around the schedules of Prince George, 12, Princess Charlotte, 10, and Prince Louis, 7. They drop their kids off at school and sit around the table for dinner each night—and although it’s unclear what the specific obligation was that made Prince William refuse to help his grandmother, Jobson wrote that she was less than pleased.

The Windsor Legacy author noted that William’s reaction “left his grandmother not only disappointed but irritated. She is said to have scoffed: ‘Isn’t that what nannies and policemen are for?'”

In the end, the “ailing Queen” went to the hospice anyway, bringing Princess Anne with her. The July 15, 2022 event was to become the last engagement Queen Elizabeth ever carried out in England.