Downton Abbey's fourth season ends with the Crawleys helping rescue the reputation of Prince Edward, the then-Prince of Wales and son of King George V. The heir's good name is put at risk after the sneaky Mr. Sampson (Patrick Kennedy) steals a letter the prince wrote to his mistress, Freda Dudley Ward (Janet Montgomery). While the love-letter storyline was pure fiction, Edward and Freda's relationship is based on the young royal's scandalous real-life romance with the married socialite, which would eventually repeat itself when he got famously entangled with Wallis Simpson.Source: ITV 
The Real Freda
Freda is often called Prince Edward's "first love," and the two carried on a long affair. The pair reportedly met in a doorway in the tiny Mayfair neighborhood of London while seeking shelter from a WWI air raid in 1918. At the time, Freda was 23 and a mother to two daughters and had been married for five years to the much older Liberal MP, William Dudley Ward. While Feda and Edward were open about their love for one another in the safety of aristocratic house parties and exclusive nightclubs, their romance was considered quite shameful at the time, since Freda was a married woman. However, the majority of the British public knew nothing of the couple's relationship, as the British press refused to write about Edward's scandalous private relationships. Despite this, gossip about Fred and Edward's romance often ended up in American newspapers, which weren't beholden to the same standards at their British peers when it came to the monarchy. This is why Martha Levinson (Shirley MacLaine) mentions that she has read all about Freda in the newspapers back in America and Rose (Lily James) is convinced that Sampson will try to sell Freda's letter to a newspaper in America: the British press wouldn't dare touch such a scandalous item.
When the season finale takes place in 1923, the two had been a couple for nearly five years. While we are never told what is in Freda's letter, we do have some insight into what they might have said. Back in 2003, decades after Freda and Edward had both passed away, over 300 letters that Edward had written to Freda between the time of 1921 and 1923 were put up for auction at Sotheby's New York. With the contents now open to anyone, the curious public has gotten a glimpse into Edward's private correspondence with Freda. The pair had cutesy nicknames (she was "Toi," he was "Moi"), and he referred to himself as "David," complained about his various royal duties as the Prince of Wales, and peppered his correspondence with misogynistic and racist language. According to The New York Times , he would describe "any woman who paid the prince unwelcome attention" as a "b*tch" and had even worse descriptions of officials he met during tours of Japan, India, and China.
Whatever was in the letters, on Downton Abbey, Rose and the Crawleys' involvement clearly means a lot to the fictional Freda and Edward, as Freda convinced the prince to give Rose the first dance at her debutante ball. A public dance like this would have been considered an immense honor at the time and would most certainly have cemented Rose's place as a the new It socialite in London society.Source: Getty 
And Then Came Wallis
There is some speculation that the two broke up around 1923 but still remained close friends. Either way, Freda was considered Edward's very close confidant until 1934, when she reportedly telephoned Buckingham Palace and was told that Edward had asked that her calls no longer be put through to him. It turns out that he had met another married woman, Wallis Simpson, and was desperately in love. He would eventually cut off all ties with Freda as he embarked on a romance with the woman for whom he would eventually abdicate the throne, as you may recall from The King's Speech. Freda divorced William in 1931 and later married Pedro Jose Isidro Manuel Ricardo Mones, Marques de Casa Maury, in 1937. The couple divorced in 1954. To her credit, Freda never publicly spoke about her affair with Edward, even after he became a disgraced member of the royal family following his marriage to Wallis.Source: Getty