News media yet again caught flat-footed as they misread America's complex political pulse

All the dazzling technology, the big data and the sophisticated modelling that American newsrooms bring to the fundamentally human endeavour of presidential politics could not save American journalism from yet again being behind the story, behind the rest of the country.

The news media by and large missed what was happening all around it, and it was the story of a lifetime. The numbers weren't just a poor guide for election night - they were an off-ramp away from what was actually happening.

No one predicted a night like this - that Donald J. Trump would pull off a stunning upset over Hillary Clinton and win the presidency.

The misfire on Tuesday night was about a lot more than a failure in polling. It was a failure to capture the boiling anger of a large portion of the American electorate that feels left behind by a selective recovery, betrayed by trade deals that they see as threats to their jobs and disrespected by establishment Washington, Wall Street and the mainstream media.

gut feeling

Journalists didn't question the polling data when it confirmed their gut feeling that Mr Trump could never in a million years pull it off. They portrayed Trump supporters who still believed he had a shot as being out of touch with reality. In the end, it was the other way around.

It was just a few months ago that so much of the European media failed to foresee the vote in Britain to leave the European Union. Election 2016, thy name is Brexit.

Election Day had been preceded by more than a month of declarations that the race was close but essentially over. And that assessment held even after the late-October news that the Federal Bureau of Investigation was reviewing a new batch of emails related to Mrs Clinton's private server.

Clinton's victory would be "substantial but not overwhelming", The Huffington Post had reported, after assuring its readers that "she's got this".

That more or less comported with The New York Times's Upshot projection early Tuesday evening that Clinton was an 84 per cent favourite to win the presidency.

Then came a profound shift, as mainstream media organisations scrambled to catch the bus that had just run them over. By 10:30pm, the Upshot projection had switched around, remarkably, to 93 per cent in Trump's favour.

Other major sites also flipped from a likely Clinton victory to a likely Trump victory. John King of CNN proclaimed to his huge election night audience that during the previous couple of weeks, "We were not having a reality-based conversation" given the map he had before him, showing Trump with a clear opportunity to reach the White House.

That was an extraordinary admission; if the news media failed to present a reality-based political scenario, then it failed in performing its most fundamental function.

The unexpected turn in the election tallies immediately raised questions about the value of modern polling: Can it accurately capture public opinion when so many people are now so hard to reach on their unlisted cellphones?

"I think the polling was a mess," Stanley Greenberg, a Democratic pollster, told me Tuesday night. "But I think a lot of it was interpretation of the polls."

Mike Murphy, a Republican strategist, said on MSNBC, "My crystal ball has been shattered into atoms" because he predicted the opposite outcome. "Tonight data died," he added.

Regardless of the outcome, it was clear that the polls, and the projections, had underestimated the strength of Trump's vote, and the movement he built, which has defied all predictions and expectations since he announced his candidacy last year.

And that's why the problem that surfaced on Tuesday night was much bigger than polling. It was clear that something was fundamentally broken in journalism, which has been unable to keep up with the anti-establishment mood that is turning the world upside down.

Politics is not just about numbers; data can't always capture the human condition that is the blood of American politics. And it is not the sole function of political reporting to tell you who will win or who will lose. But that question — the horse race — has too often shadowed everything else, and inevitably colours other reporting, too.

You have to wonder how different the coverage might have been had the polls, and the data crunching, not forecast an almost certain Clinton victory. Perhaps there would have been a deeper exploration of the forces that were propelling Trump toward victory, given that so much of his behaviour would have torpedoed any candidate who came before him.

global stocks

Maybe we'd know a lot more about how Trump's plan to build a wall along the southern border would fare in Congress, or what his proposal to make it easier to sue journalists might actually look like. How about his plan to block people from countries with links to terrorism?

Then there was the drop in the global stocks on Tuesday night, which wasn't just figures on a screen but wealth being erased. The expectations were out of whack, and Wall Street doesn't do out-of-whack well.

What's amazing is how many times the news media has missed the populist movements that have been rocking national politics since at least 2008. It failed to initially see the rise of the Tea Party, which led to the Republican wave of elections of 2010 and 2014, which was supposed to be the year the so-called Republican establishment regained control over its intra-party insurgency.

Then, of course, there was Trump's own unexpected rise to the nomination. In their view the Government was broken, the economic system was broken, and, we heard so often, the news media was broken, too. Well, something surely is broken. It can be fixed, but let's get to it once and for all.