A slight correction:
I inferred in this week's show that the very first Supreme Court -- with Chief Justice John Jay -- met in Federal Hall. They actually first convened on February 2, 1790, in a building very close by to Fraunces -- the Royal Exchange Building. Also called the Merchant Exchange, the Court's first home was located at Broad and Water streets, making it practically Fraunces' neighbor. At the time there were only six justices that served on the court.
It was completely unsuited for such important work. According to writings from 1920 by Joseph Bucklin Bishop, the Exchange was "a very curious structure, for its ground floor was open on all sides, and in tempestuous weather the merchants who gathered there for business found it extremely uncomfortable. It had a second story which was enclosed and consisted of a single room" [source] Here's an illustration of this odd building:
By 1791, the court moved to Philadelphia. A more dignified Merchants Exchange was later built in New York and featured a well-regarded statue of Alexander Hamilton in its rotunda. Unfortunately this building was promptly burned down in the Great Fire of 1835.
So, is Fraunces Tavern really the oldest building in Manhattan? It really depends on how much leeway you're willing to give it. There's been a continually standing structure there since 1719, easily outdistancing two other Manhattan buildings, St. Paul's Church on Broadway and Fulton streets (1766) , and the Morris-Jumel Mansion in Washington Heights (1765).
Fraunces, however, has gone through a host of radical changes to its appearance, with floors added and removed, its rooms reconfigured and its exterior entirely altered as to render it almost unrecognizable. A renovation in the 1900s by architect William Mersereau did bring it closer to its original state. There are certainly elements from the original structure that remain. Is that enough to bestow it the title Manhattan's oldest building?
There are several buildings in Staten Island, Brooklyn and Queens that lay claim to being much older. You can read about some of them here.
Downstairs at Fraunces:
You can read the story about the alleged 'dungeon' underneath Fraunces Tavern here: MAY HIDE DARK SECRET OF FRAUNCES'S TAVERN; Proprietor Likely to Conceal Noisome Dungeon from "Blisters." SO HE TERMS THE BUYERS
Places to Visit:
You can find directions and hours to the Fraunces Tavern Museum here. We recommend hanging a right at the second floor and watching the short introductory video before exploring the room.
When you're done with the museum, head on up Pearl Street one block north to see some curious ruins under foot, the remnants of old Lovelace's Tavern. Bricks embedded in the sidewalk also indicate where the Stadt Huys (or New Amsterdam's city hall building) once stood.
Pearl Street sat along the edge of Manhattan in the 1660s, meaning the land Fraunces sits upon today would have been water and docks. This interactive map from PBS's Dutch New York display illustrates this pretty effectively.
Curious to learn more about New York during the Revolution? Check our two-part podcast series from 2008: The British Invasion: New York 1776 and Life In British New York 1776-1783.
There's not any real contemporary books on Fraunces Tavern history, but you might find this artifact from 1919 of interest -- A Sketch of Fraunces Tavern and Those Connected With Its History, a short 'official' history by Henry Russell Drowne, a member of the Sons of the Revolution.
Drowne was best known as a collector of coins and printed money but was active in New York historical preservation as well. In stark contrast to his name, Drowne died in a house fire on the Upper West Side in 1934.