George Washington had his Martha, John Adams his Abigail, and when most people think of James Madison, it is the famous Dolley who usually comes to mind. But there had been another woman in Madison’s life long before he met the widow Dolley Payne. Her name was Catherine Floyd, daughter of Colonel William Floyd of New York, and she broke Madison’s heart.

In 1783, while he was in Philadelphia serving as a member of the Virginia delegation to the Continental Congress, Madison fell in love with Miss Kitty. He was at the time thirty-two, and Kitty only sixteen, but this seemed to be no obstacle to their courtship. In April of that year, Madison wrote to his good friend Thomas Jefferson, “Since your departure the affair has been pursued. Most preliminary arrangements although definitive will be postponed until the end of the year in Congress.” Anticipating their forthcoming nuptials, the couple exchanged miniature portraits of themselves painted by Charles Willson Peale, and when the Floyds left Philadelphia to return to New York, Madison accompanied them as far as New Brunswick.

Madison returned to Philadelphia assured of Miss Floyd’s affection, but not long afterward received a letter from her abruptly breaking the engagement and returning his miniature. She had spurned the shy, solemn Virginian for a nineteen-year-old medical student at the College of Philadelphia, William Clarkson, whom she married in 1785.

Madison was miserable at the time and wrote to Jefferson about his “disappointment.” Jefferson returned a consoling letter, reminding Madison that “the world still presents the same and many other resources of happiness.” Despite Jefferson’s sage advice, it took Madison eleven years to recover from his blighted romance. He was forty-three when he married Dolley Payne; nevertheless, he enjoyed forty-two years of happiness with her.

Though Madison and Kitty Floyd went their separate ways, the Charles Willson Peale miniatures of them have been reunited in a special display at the Library of Congress.