You either love or hate Baz Luhrmann’s unabashedly lavish 2013 film adaptation. But one thing’s for sure: few works capture the American imagination more than F. Scott Fitzgerald’s The Great Gatsby.

It was the Roaring Twenties, an era when flapper dresses and fireworks set to brass jazz bands sent nights spiraling into debauched infamy. It was a world we now can only imagine, but once, long ago in the glimmering past, it did exist. And though Luhrmann's set designer Catherine Martin built Jay Gatsby's mansion on soundstages for the film, all that jazz was based on Long Island's real architectural flair.

The North Shore "Gold Coast" on the Long Island Sound boasts a history of affluence, with private estates of Vanderbilts, Roosevelts, and others of Gilded Age fortune built there at the turn of the 20th century. The parties soon followed. Fitzgerald was invited.

Myths abound on which mansions inspired the Gatsby castle and the Buchanan home. Is it the legendary Beacon Towers? The fabled Land's End? Let’s start, as the novel's Nick Carraway did, with Tom and Daisy Buchanan.

If there was such a home for the Buchanans, it likely looked something like the Old Westbury Gardens, ten miles from Sands Point. In 1903, John Shaffer Phipps, heir to U.S. Steel fortune, sought to woo his British fiancée Margarita, promising to build her a home in likeness of her own family home in Battle Abbey.

Phipps, called (coincidentally) "Jay" by his friends, enlisted the English designer George A. Crawley and furnished the chambers with the finest English antiques. The Charles II-style mansion nests among 200 acres of prim gardens, woodlands, and ponds. It was added to National Register of Historic Places in 1976. The now commercial property welcomes visitors for tours, exhibits, and events.

But the main event was—fictitiously, at least—across the bay: the legendary mansion of the great Gatsby himself. In contrast to the “old money” demure of the East Egg estates, Gatsby’s abode as Fitzgerald painted it was the expression of extravagance.

The Oheka Castle was one such colossal affair. The name is an abbreviation of its owner's, financier Otto Hermann Kahn. Kahn, born in 1867 in Mannheim, Germany, rose in the ranks of the Deutsche Bank before moving to New York and marrying Addie Wolff.

One day, the couple set off to build themselves a summer home—and the second-largest private residence ever built in America.

Finished in 1919, the main building's 127 rooms sit amid a 443-acre plot. The house, designed by Delano & Aldrich, was built on an artificial hill, giving it a splendid view of the Cold Spring Harbor.

Nothing was too great for Kahn. For the grounds, he commissioned none other than the Olmsted Brothers whose illustrious landscape resume is too long to list. The centerpiece was a French-mannered, axial sunken garden, with water terraces, hedges, and parterres leading to the grand entrance.

The complex, like Gatsby’s, was completed by a swimming pool—that, and an 18-hole golf course, tennis courts, orchards, greenhouses, stables, and a landing strip.