Their names were Shepard Mallory, Frank Baker, and James Townsend. Little evidence documents their existence as enslaved field hands on a farm near Hampton, Virginia, aside from a few scribbled notes in an overseer’s journal, listing them as property. But their lives were forever changed after the first shots of the Civil War were fired at Fort Sumter on April 12, 1861. The three men were taken by their master to Sewell’s Point, near present-day Norfolk, and put to work building an artillery battery for the Confederacy. Within days, they learned that their owner wanted to take them to North Carolina, where, separated from their families, they would be put to work constructing another rebel output.

They had a decision to make: Go with their master and aid the Confederate war effort? Or embark on a risky run for freedom by escaping to the Union stronghold at Fort Monroe? Mallory, Baker, and Townsend knew that if they were turned back at the fort, they would likely face whipping—or worse. They also knew that if they somehow persuaded the Union soldiers to offer them refuge, their families in Hampton might be harmed in retribution. They didn’t know that the consequences of their decision would mark the turning point in a war, long before the battles of Vicksburg or Gettysburg.

On the evening of May 23, as Confederate sympathizers celebrated Virginia’s decision to secede, the three men made their move, rowing a small boat across Hampton Roads to Fort Monroe, one of the only Union-controlled outposts in the South. The fort’s commander, Gen. Benjamin Butler, was no abolitionist—he had voted for Jefferson Davis at the 1860 Democratic National Convention. And Union policy on slavery was clear: President Abraham Lincoln maintained from the outset of hostilities that he had no intention of interfering with the “peculiar institution”; rather, the Union’s aim was to crush the Southern rebellion.

Nevertheless, Butler realized the absurdity of honoring the Fugitive Slave Law, which dictated that he return the three runaways to their owner. They had been helping to construct a Confederate battery that threatened his fort. Why send them back and bolster that effort?

So the general struck upon a politically expedient solution: Because Virginia had seceded from the Union, he argued, he no longer had a constitutional obligation to return the runaways. Rather, in keeping with military law governing war between nations, he would seize the three runaways as contraband—property to be used by the enemy against the Union.

Lincoln let the decision stand; Butler, after all, hadn’t challenged the status of enslaved people as property. Yet Mallory, Baker, and Townsend’s escape and the general’s clever gambit proved momentous, the repercussions heralding the beginning of slavery’s end. For when other enslaved African Americans heard that three men had been granted refuge, they began flocking to Freedom’s Fortress, as they called Fort Monroe. They came despite rebel rumors that the Yankees would eat them, sell them into slavery in Cuba, process them into fertilizer, or make them pull carts like oxen.

By war’s end, approximately half a million formerly enslaved people and other African American freedmen had sought protection behind Union lines. These “contraband,” as they became known, usually lived in camps hastily erected almost anywhere the army was stationed. The large number of runaways who flocked to Union lines belies the outdated and racist notion that enslaved African Americans simply waited for emancipation by singing hymns and strumming banjos; rather, they seized almost every chance to pursue their freedom, often risking death, and in so doing, helped make slavery a central issue of the Civil War.

“Where we are drifting, I cannot see,” wrote abolitionist Lydia Maria Child soon after Mallory, Baker, and Townsend made their historic escape, “but we are drifting somewhere; and our fate, whatever it may be, is bound up with these … ‘contrabands.’” Indeed, as runaways began inspiring hope among the enslaved, the issue quickly made national headlines, sparking a debate about what to do. Some Union generals, including George McClellan, maintained that they would not interfere with slavery and offered no refuge for the runaways. The question of what to do with the refugees gained increasing importance as the Union army pushed into the Deep South and discovered massive plantations abandoned by their Confederate owners. A chaplain with the Army of the Tennessee described how African Americans “flocked in vast numbers—an army in themselves—to the camps of the Yankees,” likening the influx to “the oncoming of cities.”

African Americans, meanwhile, discovered that conditions in a contraband camp, depending on who was in charge and where it was located, were often worse than life on the plantation. Refugees in these makeshift villages of tents or shacks sometimes suffered abuse at the hands of Union soldiers, faced reprisals when Confederates infiltrated Union lines, were occasionally handed over when their owners came looking for them, and fell victim to smallpox and diseases that proliferated in the absence of proper sanitation. Increasingly, women and children arrived in the camps—barefoot and hungry after long journeys by boat, wagon, foot. “The suffering from hunger and cold is so great,” wrote a Union commander in Tennessee, “that those wretched people are dying by scores … sometimes 30 per day die and are carried out by wagon loads, without coffins, and thrown promiscuously, like brutes, into a trench.”

The president’s position on slavery had slowly evolved, and eventually he concluded that the Union could not win the war without destroying the institution. On Jan. 1, 1863, his Emancipation Proclamation, which freed enslaved people in territory under rebel control, also decreed that African Americans could enlist in the Union army.

Union generals and troops had realized from the beginning how African Americans in contraband camps could aid the war effort, especially by undertaking tasks white soldiers were loath to do. (Whether the wages the Union promised were actually paid was often another matter.) Women made significant contributions, growing cotton on abandoned plantations as part of a Union labor program, for example, or becoming cooks or hospital workers. And many runaways contributed valuable military intelligence, reporting on Confederate troop movements when they arrived at Union lines. George Scott, who escaped to Fort Monroe, became a spy and bragged that he could “smell a rebel further than a skunk.” Harry Jarvis, one of the early arrivals at Monroe, had predicted that African Americans would serve on the front lines. He had asked Butler to enlist him in the army but was told “it wasn’t a black man’s war.” Jarvis responded that “it would be a black man’s war before they got through.” He eventually joined the 55th Massachusetts Infantry Regiment and lost a leg in the battle of Folly Island.

Lewis Lockwood, a member of the American Missionary Association, a New York City-based abolitionist group that aided the afflicted in contraband camps during the war, had made a prophetic prediction on arriving at Fort Monroe in September in 1861: “On the contraband under God perhaps hinges the destiny of this Republic. Without them this rebellion may not be suppressed.”

The rebellion was suppressed. And on July 4, 1865, in a ceremony on Roanoke Island, the site of a contraband camp-turned-freedmen’s village, a sergeant in the 27th Colored Troops reflected on a day when “all our race … would be able to dwell under the bright and genial rays of universal liberty.”

Today, a monument on Roanoke Island, established a decade ago by the National Park Service with the help of descendants of the freedmen, pays tribute to the African Americans who cast off the shackles of slavery. It’s one of the few memorials that celebrate contraband heritage, though in recent years long-forgotten vestiges of these sites—camps, cemeteries, freedmen’s villages—have been preserved. In Corinth, Mississippi, the Park Service and a local organization established a memorial park on the site of a former camp, with bronze statues that portray the contraband experience: African Americans learning to read, taking up arms, tending to crops. A ceremonial Freedom Park was established in Helena, Arkansas, near the site of a camp whose occupants helped build nearby Fort Curtis in 2013.

In Alexandria, Virginia, two self-described “little old ladies” have helped preserve a cemetery where both contraband and freedmen were buried—a campaign that began as something of an accident. Louise Massoud and Lillie Finklea met in 1997 when they joined other residents to fight the proposed expansion of the Woodrow Wilson Bridge across the Potomac. Thanks to a local historian, they discovered that the redevelopment plan threatened a burial ground established by the federal government in 1864 for the African American refugees around the nation’s capital who had succumbed to disease. The site was used for five years, then slowly faded from collective memory. In the 1950s, a Mobil gas station and an office building were constructed on the sacred ground. After the proposed bridge project was announced a half-century later, archaeologists using ground-penetrating radar confirmed that at least 500 of the 1,800 original graves survived—a discovery that ultimately saved the site from destruction.

For Finklea, the cemetery represented an important connection to the once-thriving African American presence in Alexandria, a reminder “that we were really here,” in a city where many links to that heritage have vanished. The two women decided to make an appointment with Mobil, which owned the property, at the company’s headquarters in Fairfax, Virginia. “If you want to know how simple-minded Lillie and I were, we drove out there to ask them to give us the land,” says Massoud. She and Finklea laugh at the absurdity, in hindsight, of the Mobil executives saying, Why sure, here’s the deed. Company officials said they’d look into the situation. They never responded. “We had the nerve, though,” Finklea says. “We weren’t afraid of anything.”

Massoud and Finklea held ceremonial wreath hangings and managed to get a highway marker erected explaining the site’s significance. When the federal Department of Transportation gave the City of Alexandria money as remediation for the bridge project, the two women helped persuade city officials to use $5.5 million of the funds to purchase and preserve the cemetery property. The gas station and office buildings were demolished, and the Contrabands and Freedmen Cemetery opened in 2014. “Don’t tell me what two little old ladies can’t do if they put their mind to something,” Finklea says.

The memorial honors those who never experienced true freedom—more than half the victims were younger than 16. And it’s where visitors can learn about the estimated 75 contraband-turned-soldiers, killed fighting for the Union, who were initially buried here rather than at Alexandria National Cemetery, where white troops were interred. Angered by this slight, 443 African American troops recovering at nearby L’Ouverture Hospital sent a letter to the surgeon in charge. “We are not contrabands, but soldiers of the U.S. Army,” it reads. “We are now sharing equally the dangers and hardships in this mighty contest, and should share the same privileges and rights of burial in every way with our fellow soldiers, who only differ from us in color.”

Their plea reached Maj. Gen. Montgomery Meigs, the quartermaster general in Washington, who ordered that the soldiers buried at the freedmen cemetery be reinterred at Alexandria National, and that all subsequent African American troops be accorded the same honor.

Today, you can walk along the grassy pathways of Alexandria National, lined with hundreds of tombstones engraved with the names of Civil War soldiers and the initials USCT—United States Colored Troops. The cemetery is less than a mile from the freedmen’s plot, but in that short distance one can see the initial stirrings of the long, arduous fight for equality.

Adrienne Washington, a writer and professor, will happily tell you about that fight at Fort Ward, just a short drive from the two Alexandria cemeteries. Fort Ward, constructed during the Civil War as one of the most significant in a chain of forts and batteries designed to protect the nation’s capital from Confederate attack, was preserved in the 1960s, with reconstructed officers’ quarters, a small museum interpreting the site’s history, and various signs along the earthworks—partially rebuilt—describing the Rodman guns and smoothbore Howitzers used by Union troops.

But Washington is less interested in discussing barracks design or battle history; this to her is hallowed ground, where her ancestors, and her neighbors’ ancestors, not only helped the Union war effort but also established a thriving African American village during Reconstruction. And she wants that heritage—long obscured and, more to the point, disrespected—to be memorialized and honored, just like the park’s other layers of Civil War history.

Washington’s great-grandfather James Montgomery Peters, enslaved on a Virginia plantation at war’s start, escaped on learning he was going to be sold to another master. He eventually joined the 1st Colored U.S. Infantry on nearby Mason’s Island, now called Roosevelt Island, in the Potomac, and was wounded at the battle of Fair Oaks.

Other escaped slaves worked at Fort Ward and the adjacent Virginia Theological Seminary (converted into a hospital during the war), eking out an existence amid widespread suffering. Jane Stuart Woolsey, the hospital’s superintendent of nurses, wrote in her memoir that “the Chaplain was called on frequently to visit and bury the ‘contrabands’ whose poor little huts hung on the edges of the camp and were scattered over the fields all the way to the City.”

After the war, in the shadow of Fort Ward, freedmen purchased small plots and established their own community, building houses, a school, a church—and more important, building lives in the aftermath of emancipation. Their settlement thrived until the 1950s, when the city acquired the land (in some cases using eminent domain) and razed all the structures, making way for the park honoring the fort’s history on the eve of the Civil War centennial.

The site’s African American history remained largely obscured, the only prominent reminder a tiny cemetery still owned by the Oakland Baptist Church. Then, two years ago, the Alexandria Department of Recreation proposed adding picnic tables and pavilions to the park’s facilities. Washington and other descendants of the original African American settlers protested that the development threatened land where their ancestors were buried. As it was, the park system had already despoiled the grave of Clara Adams, Washington’s great-great-aunt, by establishing a maintenance yard around her headstone many years ago.

The city eventually hired Tom Bodor of the Ottery Group in Olney, Maryland, to excavate suspected burial sites. He discovered or confirmed 22 graves, some adjacent to the fort, and also unearthed a privy dating to the 1900s. Bodor says that the preliminary excavation was promising enough to suggest that further work at the site could reveal undisturbed Civil War-era artifacts. Meanwhile, the Office of Historic Alexandria secured a $15,000 grant from the National Trust’s African American Historic Places Initiative to erect signs describing the history of the freedmen’s settlement in the park. And the site’s museum has incorporated exhibits about the African American troops during the way and the story of the contraband.

It's a start, but Washington and the other descendants want the gravesites protected. “They took this land from us 50 years ago, Washington says. “All we ask is, give us back an opportunity to restore and preserve our heritage, so we can pass that along, not only to our children, but to all children. One of my biggest hopes is that with this park, we will be able to represent what has happened to African Americans all over the South, where people contributed to their own freedom, either by joining up with the troops, or somehow contributing to the contraband effort.”

It's a story that starts, of course, back at Fort Monroe, where we can mark the beginning of the end of slavery in America. Yet Fort Monroe is also where we can mark slavery’s inception. For in 1619, a Dutch corsair arrived at Point Comfort, which overlooks the confluence of the Chesapeake Bay and the James River, carrying 20 Angolans captured from a Portuguese slave ship—the first recorded instance of enslaved people alighting in the New World. Two hundred years later, enslaved people helped construct Fort Monroe a few hundred feet from Point Comfort, not knowing that within a matter of decades, Mallory, Baker, and Townsend would pass through the main gate of Freedom’s Fortress.

“This is the only place in America like it, in the context of African American history,” said Robert Engs in a 2011 interview. Engs, who died in 2013, was an emeritus professor of history at the University of Pennsylvania and author of Freedom’s First Generation, a landmark study about emancipation in Hampton. He argued that African Americans’ resistance to slavery during the Civil War represented a “Great American Slave Rebellion,” an idea first advanced by W.E.B. Du Bois. Of course, Engs acknowledged, the rebellion depended on the power of the Union: When the army retreated in Alabama and Tennessee, for instance, runaways in contraband camps were reclaimed by their masters or once again sold into bondage. Yet without the contributions of African American soldiers and laborers, the Union would have lost the war, he said. Fort Monroe, where the rebellion for freedom began, represents “black America’s Lexington and Concord, our storming of the Bastille.”

Engs would tour the fort with other members of the Hampton-based Contraband Historical Society, including its founder, Gerri Hollins, whose great-great-grandmother Emma Ann escaped from slavery and found refuge at Fort Monroe during the Civil War. Hollins, who died in 2012, pushed for Hampton to create a museum or major memorial honoring the contraband camps.