Sag Harbor, tucked into the South Fork on the easternmost end of Long Island, has always maintained its own unique identity amid the upscale allure of the Hamptons, thanks in part to its rich history. The community of Eastville was a prominent free African American settlement in the early-to-mid-1800s, a draw for adventurous men who wanted to try their luck at the whaling industry. Almost a hundred years later, starting in the late 1940s, the area began to enjoy a renaissance as an African-American vacation hotspot.

Colson Whitehead, in his 2009 novel Sag Harbor, describes the summer social scene among African-American vacationers as intimate and familiar, where everyone served as members of an extended family. “The middle-aged ladies camped out in front of someone’s house each afternoon, usually ours, as folks from Azurest and the Hills and Ninevah promenaded by, making the rounds, leaving footprints that were physical traces to a dozen conversations,” he writes.

The areas that Whitehead references—Azurest, Sag Harbor Hills, and Ninevah—are the historically African-American beach communities in Sag Harbor, with a long tradition of vacation-home ownership passed down through generations. The area has also traditionally attracted vacationers from the upper echelons of African-American society—including magazine publisher Earl Graves, restaurateur B. Smith, and former editor-in-chief of Essence magazine Susan L. Taylor. Both Langston Hughes and Colin Powell spent time there in their childhoods.

“[This community] started out with a few intimate people who were friends of friends and relatives of friends, so to speak,” says Dianne McMillan Brannen, a real estate agent specializing in properties in Azurest, Sag Harbor Hills, and Ninevah. “Word-of-mouth is still a very important element in gaining access to this community.”

The demographics of Sag Harbor have been shifting somewhat in recent years—real estate agents estimate that about 30 percent of the residents of the beach communities are not African American, which is a dramatic increase from past decades. Children of many of the original owners may have difficulty absorbing the cost of a vacation home in the Hamptons, leaving more opportunities for outside brokers to sell the homes. There has also been an increase in year-round residents.

“Many of the other developments are more integrated now, more so than they were way back in other years,” says Kathy Tucker, 87, who spent summers in historic Eastville before retiring there full-time in 1984. As a member of the Eastville Historical Society, she’s been involved with excavating the history and lineages of the families who lived in the community during its whaling heyday.

While the private beach enclaves of Sag Harbor are in flux, Brannen says that anyone who appreciates the beauty and history of the area is a welcome addition.

“Sag Harbor is the last of the Mohicans in terms of a beautiful seaside resort,” she says. “We all have the same common denominator of loving the area. Everybody and anybody who is happy to be here and preserve the community is a welcome body.”