The District didn’t always resemble a piece of squareish bread with a huge bite-size chunk missing. Once upon a time—more than 160 years ago—Arlington County and part of the city of Alexandria fell within the District’s limits, completing a full diamond that spanned the Potomac River.

Or at least it did until 1847, when the federal government retroceded those territories back to Virginia. Along with Maryland, the commonwealth had forfeited territory in January 1791 when lawmakers implemented the Residence Act, creating a national capital city overseen by Congress and comprising a total of 100 square miles, 31 of which came from a sliver of the Old Dominion. Officials had settled on the area after previously considering already existing cities like New York and Philadelphia.

Using the Potomac as a boundary, planners divided the District into two counties: Washington to the north and Alexandria to the south. The latter included much of what today we call Old Town Alexandria and Arlington County.

Taken altogether, the new federal jurisdiction was incorporated 10 years later with the passage of the Organic Act of 1801.

Alexandria earned status as an independent city—that is, a city governed separately from a county or any other larger local jurisdiction—in 1870. What remained of Alexandria County would eventually become Arlington County in 1920.

As with most real estate issues in D.C., things soon became complicated. Talk about returning the land began immediately after the District was created, and the push to do so gained momentum in the mid-1800s. By then, northern Virginia had become a major center of the slave trade, with as many as a thousand slaves passing through the Alexandria market every year.

Rumors that the capital city would abolish slavery—confirmed with the passage of the Compromise of 1850—frightened those in the business. Furthermore, concerns about a growing abolitionist movement in Virginia itself had the pro-slavery contingent pulling for the two extra votes that Alexandria County would bring to the state’s General Assembly.

Furthermore, Congress never really ended up using land on the Virginia side of the river. The Alexandria port faced competition from Georgetown, whose charter as an existing city had been left intact, and citizens of the county felt neglected. From 1840 to 1846 there was a sustained lobbying effort to press Congress to return Alexandria to Virginia.

The state government welcomed the idea; thinking the issue to be pretty clear-cut, Congress approved the retrocession on July 9, 1846. President James K. Polk signed it into law the next day.

Surprisingly, a referendum that September found that while most people living in the city of Alexandria supported the law, a majority of those in Alexandria County did not. This prompted several more months of debate, and the Virginia General Assembly didn’t accept the measure until March 13 of the following year.