Westerman heads into the wind and one-man crew David Letson drops the anchor. The other three passengers and I plunge into the glowing blue water. As we stand neck-deep off the island’s white sand beach, Westerman flings a volley of ripe mangoes into the water beside us. David then teaches me his form of aquatic limin’, local slang for “chilling,” which he claims to have invented: Lie back in the sunshine, never exhale more than a half breath, and float with “absolutely no effort.”

Physically and spiritually buoyant, I decide to keep my salty, mango-drenched lips sealed about how I had independently perfected this technique 15 years ago. That was during my one-year stint as a scuba instructor 40 miles away on St. Thomas, which together with St. John makes up the rest of the major U.S. Virgin Islands.

All three offer the paradise exemplified by Buck Island, but St. Croix’s inhabited monuments—brick and-mortar links to its past—are a unique, defining asset. And as Westerman makes clear when I ask how he got into sailing, the islanders carry their own connections to history. As a kid on Nevis, a few islands down the chain, he would play sailor with his brother on the stone stairs of a ruined seaside house. “We didn’t have any idea those were the steps to the house where Alexander Hamilton was born,” he recalls.

That’s the Alexander Hamilton, the one on the $10 bill. Like Westerman, the Founding Father was a Nevisian transplant to St. Croix. The future architect of the American financial system cut his commercial teeth in Christiansted, then the booming capital of the “Danish Islands in America.” If Hamilton were alive today, he’d recognize the place.

The people of St. Croix, or Crucians (pronounced KROO-zhunz), like to say seven different flags have flown over their island. The first in that parade would be Spanish, because Columbus visited, claimed, and named “Santa Cruz” on his second voyage. But it was the Danish, represented by the sixth flag, who left the deepest physical imprint, adding their architecture to today’s rich melting pot of African and Caribbean culture.

When the Danes took possession of the island from France in 1733, St. Croix (Frenchified Santa Cruz) had been mostly deserted. To help attract the planters and traders necessary for a successful colonial enterprise, the island’s first governor envisioned a stately town on the northern coast that would rival Oslo, then a part of Denmark. Dubbed Christiansted (“Christian’s Place”) in honor of the reigning Danish king, the waterfront settlement was built on a strict, orderly grid.

That plan laid the groundwork for Christiansted’s most distinctive built feature: arcaded sidewalks. The Enlightenment-inspired building code allowed the second floors of townhouses to be pushed out over public streets, sheltering passersby from the sun and rain in neoclassical splendor. Steep roofs, a Northern European tradition, fortuitously thwarted hurricanes. Because of the island’s remoteness, building materials were often restricted to sources at hand, such as the pastel yellow bricks that served as ships’ ballast. Plantation owners and merchants of various nationalities constructed homes and stores in town to conduct the business of the sugar trade: exporting hogsheads of sugar or molasses, importing enslaved workers to harvest the cane and boil its juice.

More than 70 percent of old Christiansted’s large masonry structures endure, thanks to the local preservation commission, which formed at the dawn of the Caribbean tourism industry in the 1950s. “They said, ‘If we don’t do this, they’re gonna bulldoze everything,’” says David Hayes, an archaeologist who serves on today’s version of the body. The current Virgin Islands Historic Preservation Commission ensures that any exterior renovations to buildings in designated districts match their historical appearance.

I walk down Company Street’s shaded sidewalk toward one building that’s never been in danger of falling down: Christiansted’s hulky Fort Christiansvaern, which anchors the town to the harbor. Its paint has been restored to the original 19th-century mustard yellow, but there is one obvious anachronism. The flag waving over the fort is the Stars and Stripes, the seventh banner to fly over St. Croix. When the U.S. purchased the Virgin Islands in 1917 with $25 million in gold, it wasn’t for the historic architecture. It was to deny Denmark’s powerful neighbor, Germany, a potential Caribbean U-boat base during World War I.

Like Christiansted, the largely flat, arable island was divided up on a grid, which separated the 300 or so rectangular plots parceled out as plantations. Danish St. Croix was a planned community—the city and island layouts were all part of the overall business plan. The Estate Whim plantation, a 280-year-old sugar plantation near Frederiksted, originally went by the more businesslike “Number 4, West End Quarter.”

Estate Whim was carefully restored by the St. Croix Landmarks Society, the preservation group that has run it as a museum for more than 50 years. My tour guide there, Margaret, tells me the plantation owner’s stony “greathouse” was literally built with sugar: The blocks in the walls, some of them cut brain coral, are fixed with mortar hardened by molasses. Enslaved workers (and, later, paid laborers) harvested stalks of sugarcane from the surrounding fields, and then fed them into the preserved windmill’s steel rollers to squeeze out the juice.

In 1848, thousands of enslaved people from western estates such as Whim marched on coastal Frederiksted, the island’s other major town. Signaled by blasts from conch shell horns and ringing plantation bells, they demanded emancipation. After observing the revolt from Frederiksted’s fort, governor Peter von Scholten announced to the crowd, “All unfree in the Danish West Indies are from today free.”

On the rampart of that imposing, blood-red fort, I watch a pair of boys take turns diving off Frederiksted pier. Today, like most days, no massive cruise liners have docked alongside it. The current trickle of arrivals, I’ve heard, suits many Crucians just fine, despite the much-needed cash the cruise ships might bring ashore. Some islanders worry such hit-and-run visits or any influx of mass tourism—wouldn’t just overlook their rich, complex culture, it would overwhelm it.

Driving back to Christiansted, I suddenly realize I’m ignoring a particularly relevant part of Crucian heritage: Virgin Islanders drive on the left, and I’m in the right lane of the highway. That night I tell bartender Eric Gauttreau how a local driver politely yielded as I bounced my wayward rental Jeep (painted bright I’m-a-tourist blue) over the median. At the open-air bar at Savant, a popular restaurant, Eric helps ease my still-frayed nerves with a tasty punch concocted from fresh watermelon juice and local Cruzan rum.

For thousands of years before any flags flew over St. Croix, it had been home to island-hopping native people. When Columbus stumbled upon the place in 1493, the dominant people on the island were the Caribs, a group that proved less hospitable than the Tainos he’d met on his first Caribbean voyage. As his landing party of more than two dozen armed men oared back to the fleet, they spied several Caribs in a canoe. The Spaniards closed in on them and were greeted by a burst of arrows. One sailor was mortally wounded.

“It happened right over here,” says Josh Torres of the National Park Service, extending his arm toward the cape on the opposite side of Salt River Bay. “It’s one of the first battles of indigenous resistance in the New World.” The Carib archers fought fiercely, but lost the lopsided skirmish. Their fate -- killed or captured -- would eventually be shared by all of the island’s aboriginal people who didn’t flee. Less than a century later, “Santa Cruz” was uninhabited.

Torres and I are standing on the point of land where the crewmen came ashore (commonly called Columbus Landing), the first and only confirmed time anyone from Columbus’ voyages set foot on what is now U.S. territory. By then, diverse native cultures had occupied villages here for more than a millennium. They left behind an accumulated trove of artifacts, including stone tools and earthenware, many of which are awaiting excavation.

The Europeans located their first settlements in the same place, and built a triangular earthwork fort at the end of the point. But if it weren’t for Torres, I’d have no idea the steep, overgrown hill we climb is one of its walls. He leads me to the worn, grassless strip that beachgoers use as a parking lot and kneels beside a palm-size puzzle of flat, brick-colored shapes. “This was one piece of pottery, and then somebody ran over it,” he says.

Torres is the cultural resource project manager of the three areas overseen by the Park Service on St. Croix: a section of Christiansted that includes Buck Island; Salt River Bay; and Fort Christiansvaern and other restored Danish buildings. The point is owned by the local government, which co-manages the site. Although everyone involved recognizes the need to block traffic and clear the brush that camouflages the earthwork fort, financial considerations, for one, complicate the picture. A new parking lot and heavy-duty landscaping cost money that no one can spare

But there’s reason to hope such funding will become easier to find. A proposal pending before Congress would designate the entire island of St. Croix as a National Heritage Area. If passed, local cultural groups would be empowered to coordinate new preservation, conservation, and heritage tourism projects, supported by federal matching funds. The recognition of St. Croix’s rich history would raise the island’s profile as a cultural destination, attracting tourism that would embrace Crucian heritage, not trample over it.

The night before I fly back to the mainland, I stop by the bar at Savant for a farewell drink. Eric pulls a bottle of Cruzan Single Barrel dark rum from the shelf. After a smooth sweetish sip, I’m glad I took it neat.

Server Nazan Aykent joins our conversation and tells me she hails from Istanbul by way of the Big Apple. I ask her how life on St. Croix compares to the city. “Time in New York passes so fast,” she says. “I’ve only been here three months and it feels like years. I think ‘island time’ is real.” I can relate. The four days I spent on St. Croix seem like two weeks. But it was still too short.