The story of New York can be told by following its rich families, who spent more than a century and a half moving north. At first they lived at the southern tip of Manhattan, where the harbor and the money were. From there they kept climbing: Washington Square, Union Square, Murray Hill, then Fifth Avenue, and finally the Upper East Side. Each generation of wealth wanted a little distance from the one before it, and every time it got that distance, the city caught up and the march started over.
And it all starts much earlier than that. In the beginning it wasn’t even called New York it wasn’t the United States, and it wasn’t really part of anyone: it was a company’s business. It was called New Amsterdam, a Dutch trading post on the tip of Manhattan that belonged to the Dutch West India Company, founded in 1624 and built almost entirely on beaver, which sold for a fortune in Europe to make hats. A fort, a harbor, low houses, and a group of merchants who cared about the land mostly for what could be shipped out of it.
The English showed up forty years later. In 1664 a few ships sailed into the harbor and demanded surrender, and the Dutch with neither the will nor the means to fight handed the city over without a shot. They renamed it New York, after the Duke of York, and it stuck. It stayed British for more than a century after that; it only stopped being British with independence, when the last British troops left in 1783 and the city became part of the new country. But the Dutch part never washed off: it stayed in the surnames, the streets, and a trader’s mentality the city never lost. Wall Street, in fact, was the wall that marked the northern edge of that little town.
Out of that world of merchants came the name that would end up defining an entire era: the Astors. John Jacob Astor arrived from Germany with almost nothing, around 1784, and made an enormous fortune in furs. But his real stroke of genius was understanding something before anyone else: New York’s wealth wasn’t going to be in goods, it was going to be in land. While others piled up businesses, he piled up Manhattan. He bought lots north of the city when they were still farmland and waited for the city to grow toward them, which is exactly what kept happening. By the time he died, in 1848, historians describe him as the richest man in the country, with a fortune estimated at around twenty million dollars of the day. They called him “the landlord of New York.”
The second half of the 1800s was even more excessive. It’s the period Mark Twain nicknamed the Gilded Age gilded on the outside, a lot more complicated underneath. In those years a handful of men were literally inventing whole industries Carnegie in steel, Rockefeller in oil, Vanderbilt in railroads, Morgan in modern banking and piling up amounts of money the country had never seen.
One reason those fortunes grew so large, as many historians read it, is that the government of the day simply wasn’t built for that much wealth. There was no federal income tax yet that didn’t arrive until 1913 no inheritance tax, which came soon after, and nothing like today’s labor laws. The money came in with almost nothing to slow it down. So while millions of immigrants arrived at the harbor looking for work, a small group of families was putting up private palaces modeled on Europe. And the stage they chose was Fifth Avenue.
What’s now an avenue of shops was then a residential street: a row of monumental mansions where the most exclusive social life in America played out. But even inside that world there were hierarchies, and nobody ran them better than Caroline Schermerhorn Astor. Everyone simply knew her as “Mrs. Astor.”
In an age with no social media, no television, no internet, Mrs. Astor was the algorithm: she decided who belonged and who didn’t. She came from the old Dutch families the Knickerbockers, as they were called, after a character the writer Washington Irving made up and she built the city’s most famous list together with Ward McAllister, a Southerner obsessed with manners who appointed himself the arbiter of taste. The list was called “the Four Hundred,” which was, as the story goes, roughly how many people fit comfortably in her ballroom. McAllister put it bluntly: outside those four hundred, you started running into people who weren’t at ease in a ballroom. Being on the list meant you’d arrived; not being on it meant exactly the opposite. She sold nothing, but she handed out something harder to come by than money: belonging.
That world cracked open at a party. The new money kept coming, and the Vanderbilts railroad rich were, to Mrs. Astor, simply too new to let in. One of them, Alva Vanderbilt, got tired of waiting for an invitation: she built herself an enormous French-château mansion on Fifth and, in 1883, threw a costume ball for more than a thousand guests, the likes of which the city had never seen. The fine print was that she left Mrs. Astor’s daughter off the list. As the story is told, the only way Mrs. Astor could get her daughter in was to send her calling card over to the Vanderbilt house a tiny gesture, almost a formality, but in that world it meant everything. It was the old aristocracy finally acknowledging the new money. After that night, New York high society no longer fit in a single ballroom.
And once again, the city started climbing. The first jump was to the 50s on Fifth around where Rockefeller Center and the flagship stores are today where the Vanderbilts put up their houses in defiance of the conventions of the day. The decisive jump came from Mrs. Astor herself: in the 1890s she left her historic house on 34th Street and moved much farther north, to a new mansion on Fifth at 65th, facing Central Park. With that move she blessed the Upper East Side as the new territory of power. The queen changed palaces, and the court followed her, from 59th Street on up.
What happened next to those two houses says a lot about how New York works. On the 34th Street site, where Mrs. Astor had reigned for years, the family ended up putting up a hotel the first Waldorf-Astoria, in its day the most famous in the world and when that hotel went out of fashion too, they tore it down to build, on that very spot, the Empire State Building. Her Upper East Side mansion, the one at 65th, had an equally curious fate: it was demolished in the 1920s, and today, on that land, stands Temple Emanu-El, one of the largest synagogues in the world. There’s a nice turn in that detail: the congregation had started decades earlier on the Lower East Side, with German-Jewish immigrants, and had been moving north like everyone else, until it ended up exactly where the queen of old society had once held court.
Meanwhile, Fifth Avenue facing the park became the most coveted residential corridor in the country. The names that fill the history books built there: Carnegie, Frick, Whitney, Vanderbilt, Rockefeller, Astor. Each mansion was a statement, and each façade quietly competed with the one next door. Almost none survived most were torn down in the 1920s, once the taxes and the apartment buildings arrived and an apartment block suddenly paid off better than a palace for one family but the prestige stayed glued to the place. Because the luxury, in the end, was never in the buildings. It was always in the location.
Maybe that’s why the Upper East Side still feels different. It isn’t just that it’s expensive New York is full of expensive places it’s that the history there is still visible. A walk up Fifth can take you from the Frick Collection to the Met, past buildings that have spent more than a century watching fortunes, fashions and generations go by. Some families have been there forever, and others arrive looking for something money doesn’t buy so easily: legitimacy.
And there’s the lovely irony in all of it. The Astors, the Vanderbilts, the Carnegies thought they were building monuments to last forever, and their houses ended up as hotels, synagogues, museums and office towers. The fortunes were split up, the empires changed hands, the city kept climbing. What they left behind wasn’t the stone, it was the map: those same Upper East Side blocks, on Fifth, facing the park, are still the city’s center of gravity, a century and a half later. You can walk through there any morning and, underneath all of it, the same thing is still there: a city that never stopped moving north, and the people who, generation after generation, tried to stay one step ahead of it.