The sound of plucked banjo strings crackled in the air, courtesy of the great instrumentalist Béla Fleck, who sat a fishing pole’s cast away on a makeshift stage. Groups of snackers added satisfied melody lines: Each savory chomp of cucumber, grilled al dente at Boudreau’s food truck, made sounds like cans of soda popping open.

“What’s really neat about this,” says Boudreau, owner of the Vine Street Café on Shelter Island, “is it brings it full circle.”

The vegetables Boudreau served at the Plant & Sing were harvested and eaten within a few hundred yards of each other, and all within shouting distance of the nearly 300-year-old Sylvester Manor home, a Georgian-style house flanked by 20th-century porches wrapped in vines of climbing roses. It is now on its way to becoming a historic landmark, one pulsing with the vibrancy of new life.

Far from being just another old building—fighting entropy, struggling to assert its relevance in a fast and forgetful world—Sylvester Manor has undergone a renaissance because of the very thing for which it was founded: food.

“We’re digging into the past to pave our way to the future,” says Bennett Konesni, a kinetic, 11th-generation descendant of the island’s founder. Konesni started the Plant & Sing fundraiser as a way to open Sylvester Manor and the farm up to the community and celebrate history, food, and culture.

In addition to this annual community event, the farm sustains itself by selling produce to markets and restaurants in and around the perimeter of this 8,000-acre island nestled in a crook of the Atlantic between the North Fork and the South Fork of Long Island. Sylvester Manor has become so integral to the area that some establishments now allow the farm to dictate their menus. Carrie Mitchum, former executive chef at Salt Waterfront Bar + Grill, a restaurant on the island, recalls purchasing some of Sylvester Manor’s autumn gourd crop and then creating a robust and tangy puree of red kuri squash, butternut squash, acorn squash, and pumpkin.

“They adhere to high organic standards,” Mitchum says. “So as their harvest changed, so did our soup.”

These annual vegetables are value added to Sylvester Manor’s perennial crop: history. The story of European settlement on Shelter Island goes back to the mid-1600s, and food is the thread that runs root to flower: from the manor’s inception as an industrial food supplier; through its years as a summer home for Eben Norton Horsford, a pioneering food chemist; to today, when Horsford’s great-great-great-great-grandson Konesni created a life, and the Plant & Sing, here.

“Members of the same family have lived here since 1652,” said Maura Doyle, Sylvester Manor’s coordinator for interpretation and preservation. “It’s very Jane Austen, something so improbably feudal.”

There is also a shadow history to Sylvester Manor, the oldest, most intact plantation surviving in the Northeast. For nearly two centuries the fruits of this land were cultivated by people who were enslaved, men and women owned as outright as its livestock. At one time, Shelter Island had one of the largest enslaved populations in the region.

“It’s very powerful,” says Doyle, who gives tours of the home. “It’s haunting.

In 1651, a man named Nathaniel Sylvester came to Shelter Island and purchased it from Englishman Stephen Goodyear for 1,600 pounds of sugar. Then, in 1652, after a court dispute, which negated Goodyear’s right to sell the land, Sylvester purchased the island again, this time from the Manhansett People, whose indigenous ancestors had probably lived there for at least 14,000 years, archaeological evidence shows. Sylvester had three business partners, one of them his brother, and they needed food—not just for themselves, but for the people they enslaved.

Sylvester’s partners ran sugar plantations in Barbados, and by establishing a farm on Shelter Island they would be able to grow food for their labor force. By 1653 enslaved labor came to Shelter Island.

Eventually the business partnership dissolved, and Sylvester came to own Shelter Island outright. When he died, he enslaved 20 people—one of the largest enslaved populations in mid-1600s New England.
“The wealth of the manor was based on slave labor, and I say that because if you have somebody else who can farm and do the work that you would do, that’s the basis of wealth,” says landscape historian Mac Griswold, author of The Manor: Three Centuries at a Slave Plantation on Long Island, which will be published in June.

Not all was oppressive in early Shelter Island history. It’s worth noting that Sylvester showed uncommon religious tolerance, and Quaker George Fox, founder of the Society of Friends, preached here twice. The island was also a temporary sanctuary for Quaker Mary Dyer, who in 1660 in Boston was later hanged for her beliefs.

In 1666 King Charles II of England declared Sylvester’s covenant house an official manor. But it wasn’t until about 50 years after Sylvester’s death in 1680 that one of his heirs built, as a country estate atop the original home site, Long Island’s oldest Georgian-style home, which still stands today.

The front of the ochre-painted Sylvester Manor, with its symmetric dormers, six-over-six double-hung windows, and black painted shutters was built between 1735 and 1740. In early 1908, white pillars were added around the porch and the Colonial Revival-style addition was tacked on to the rear of the home.

The 18th-century part of the house features four square rooms furnished with sofas, tables, paintings, and desks. In one room, wallpaper depicting tropical palms dates to 1888 yet retains its vivid color. Across the entryway, in the East or Morning Parlor, some of the walls’ cream-colored paint from the late 1800s has chipped away, showing underneath the original cool Prussian blue from the mid 1700s. The fireplaces in both rooms are edged with Dutch Delft tiles set in 1908.

Through the years, Sylvester Manor—which once spanned the entire island—sold off most of its acreage. Ultimately the town of Shelter Island was incorporated, and today it boasts a year-round population of about 2,500 with a seasonal population almost four times that number.

Sylvester Manor freed its last enslaved person in 1821, a man named London, almost 170 years after the first enslaved men, women, and children were brought to the island and six years before slavery was abolished in New York State. It ended a shameful chapter in the manor’s history, but it also heralded the land’s shift away from food production.

In the mid 1800s, a food chemist named Eben Norton Horsford and his wife, Phebe, a lineal descendant of Nathaniel Sylvester, turned the estate into their summer home. Horsford, a Harvard University professor, perfected the recipe for baking powder by adding calcium biphosphate. He co-founded Rumford Chemical Works, an offshoot of which still produces Rumford Baking Powder. While Horsford fine-tuned his recipes in a laboratory, Sylvester Manor’s two-acre garden was adapted into the Colonial Revival style, filling it with plants such as boxwood, which is believed to have been introduced to North America by Nathaniel Sylvester himself.

Horsford’s daughter Cornelia used her inherited wealth to hire Beaux Arts architect Henry Bacon to build the Colonial Revival addition onto the original home in 1908. Bacon went on to design the Lincoln Memorial in Washington, D.C. One of his flourishes at Sylvester Manor was a musical alcove that would enhance the acoustics of stringed instruments.

By the middle of the 20th century, Sylvester Manor passed into the hands of a man named Andrew Fiske, a 9th-generation descendant of Nathaniel Sylvester. By then, the estate was whittled down to just 260 acres.

In the 1950s, a team of tree-planters dug up something amazing. They found, about 100 feet from the front door, a cannon dating to 1670 that was buried in the ground, probably to keep it out of the hands of the Dutch. The find hinted at untold treasures on the property.

Fiske died in 1992. His widow, Alice Fiske, a kind, eccentric, elderly woman who was fond of Italian hats and white cotton gloves, endowed a research center at the University of Massachusetts Boston to send down teams of archaeologists. It was a perfect match: a curious patron and hungry students excited by the good fortune of being granted access to a rare Northeastern estate that hadn’t been covered by layers of development.

“I love farming and I love being on the land,” Konesni says, and under his leadership, Sylvester Manor has begun to cultivate the next generation of American farmers. A handful of interns live upstairs in the historic house, spending their days working the soil outside. These young agrarians have come from all over the world, traveling through Gatsby country and hopping aboard the ferryboat at land’s end for the ride to Shelter Island.

“This place brings in so many special people who want to be part of its history,” says Cassie Woolhiser, 23, who hails from Missoula, Mont., and describes the experience of being a Sylvester Manor farmer as like “living in a museum.”

Where once there was forced labor, today it is largely volunteer.

Ostby and the board of directors that oversees Sylvester Manor have worked to protect the farmland by legal means, such as securing conservation easements through a local land trust. Konesni, meanwhile, threw his boundless energy into attending community meetings and bringing samples of his vegetables to area restaurants. There he won over disciples like local French chef Martine Abitbol.

“I cannot cook it if it’s not fresh from the farm,” Abitbol says.

Drawing from his year abroad, Konesni has made music a part of life at the manor. He organizes traditional and bluegrass music concerts in the Bacon-designed living room, and has taught interns to sing Ghanaian folk songs in the vegetable patches.

Every October, food and music mix at Konesni’s pièce de résistance, the Plant & Sing festival. The estate’s gates swing open, musicians perform on a harbor-side stage, poets read on rosy porches, historians give tours, and local chefs cook produce picked from the fields. Far below the harvest moon at night, farmers and friends do-sido on the lawn to the sound of a live string band.

“It’s an honor to explore Sylvester Manor in new ways,” says Konesni, dressed in blue jeans and red flannel, as he lays down his guitar between songs to take a break.

“There’s such a rich tapestry of food history here, it makes sense that we add onto it in our own way.” Suddenly, he stops. He cocks his head to make out the next song. Then he apologizes.

“I’ve got to play this tune,” he says, and he leaps back onstage to sing.