Barack Obama’s re-election has prompted a lot of shock among American conservatives, convinced they were about to get their man back in the White House. In some of the rancorous fallout, Mitt Romney’s ground game software, Project Orca, has come in for some serious criticism – this Ars Technica piece details how poorly it performed. But for anyone who has read Sasha Issenberg’s terrific look at the growing field of microtargeting and data-driven politics, it’s really no surprise that Obama had a massive advantage in getting out his vote, and that Romney failed to make up the difference in just a few months. Here’s an interview I did with Issenberg from a month ago where he reveals how Obama’s team has changed the science of winning elections forever.

From The Irish Times, October 11th, 2012

The US presidential election is the world’s most keenly followed political soap opera, and this year’s bout is proving no exception. But beneath the surface of this seemingly conventional campaign a revolution is occurring, shaping the contours of the race in ways that are barely perceptible to outsiders but which is likely to change the nature of electioneering forever.

On both sides of the aisle, armies of statisticians and number crunchers are transforming the art of campaigning into a science. Huge amounts of data is being processed and a wide array of experiments are being undertaken, all with the aim of bringing empirical rigour to a field that has long relied on hunches and intuition. Disciplines as diverse as behavioural psychology, econometrics and data analytics are being combined to more accurately target and influence potential voters. Welcome to the era of data-driven politics.

US journalist Sasha Issenberg has detailed this largely unheralded upheaval in his fascinating new book, The Victory Lab. It is rather snappily being described as Moneyball for politics, and it does share some characteristics with Michael Lewis’s celebrated look at how statistical analysis transformed baseball. However, instead of analysing baseball players, the statisticians and social scientists that Issenberg profiles are analysing both the electorate and electioneering itself, with potentially dramatic results.

“All the interesting moments occur when the rather insular world of thinking about campaigns becomes shaped by interesting thinking and problem-solving from outside campaigns,” says Issenberg over the phone, describing the genesis of the revolution. The Victory Lab sketches the history of such innovations, but pinpoints the George W Bush re-election campaign of 2004 and the groundbreaking Obama campaign of 2008 as watershed moments. “In 2004, it was the Bush world looking to the commercial world both for data and ways of thinking about marketing,” he says, “and 2008 I think the Obama campaign was really shaped by expertise from academia and technology.”

That external expertise naturally took a fresh approach to the traditional problems that every campaign faces. In terms of the fabled “ground game”, the two big issues facing US political campaigns has long been determining who to target, and how. Identifying those voters who can be persuaded to vote for your candidate, and identifying those who are definitely going with your rival, means you can be much more efficient in targeting mail-outs, email blasts and canvassing efforts. And determining what issues in particular will resonate with any given voter can allow you to refine your message to a remarkable degree – an Obama for America fundraising email from last May came in 11 different varieties, each tailored for the individual recipient. How did they decide which voters would receive which mailout?

This is where so-called microtargeting comes in – there is a vast and growing industry in the US that collects data on citizens, culling information from public records, such as voter registration records and gun licence applications, as well as private databases owned by direct mail marketers, magazine publishers, “loyalty-card” operating retail giants and the financial industry, which assiduously tracks credit scores. (This is not to mention the goldmine that is Facebook, which will undoubtedly play an increasingly prominent role in the data-harvesting trade in years to come.) Collating and analysing this data, campaigns can search for patterns and correlations, allowing skilled analysts to extrapolate a frighteningly accurate picture of any given voter’s values, priorities, even their likely level of political engagement. The result is the ability to identify the interests of individual voters and tailor campaign messaging specifically for each one of them.

“At the core of microtargeting, and what the Obama campaign really refined, is looking at the interplay between all these thousands of variables, and using it to arrive at a unique prediction for each voter’s behaviour,” Issenberg explains. “What the Obama campaign did was, through these scores, come to a prediction of your likelihood as a voter of turning out to cast a ballot at all, or of supporting Barack Obama, or holding a particular position on abortion, for instance. By updating those weekly, they were able to have a really molecular interaction with the electorate.”

The effect was largely invisible to those of us following the race between Obama and John McCain, but it was nonetheless tangible. “I quote Ken Strasma, who was Obama’s top microtargeting consultant in 2008, and he said ‘We knew who people were going to vote for before they did,’” says Issenberg. That’s an unimaginably powerful tool for a political campaign.

But the new approach doesn’t end there – the 11 different email variations sent last May were not just tailored for each voter, they were also most likely contributing to an ongoing experiment testing the efficacy of various topics and phrases. After all, what good is it to identify likely or possible voters if you can’t also identify the most effective means of persuading voters, encouraging participation and donations, and boosting turnout among your likely supporters.

“Campaigns now have tools to confidently assess cause and effect through the use of field experiments,” says Issenberg. “Our understanding of what works and what doesn’t used to be reliant almost entirely on who is the best storyteller. Because there is so much going on in a campaign in a given moment that we in the press or the public have trouble understanding what was moving votes. The use of field experiments, which are basically drug trials but which are using voters as guinea pigs, have given campaigns tools to actually measure movement in voters’ opinions and behaviour in the real world.”

To this end, an entire body of research has sprung up to rigorously test and measure the efficacy of different approaches, from robocalls to TV ads to knocking on doors. For instance, it has been demonstrated that a letter thanking someone for voting in the past can boost turnout by more than 2 percentage points; and a knock on the door from a canvasser is far more effective than a call from a phone volunteer.

“When put together the campaigns have far better tools for sorting through the electorate, and knowing whom they should engage and when, and because of experiments, have much better confidence that they know how to engage them in ways that will actually impact their behaviour,” says Issenberg.

How likely is it that these tools and techniques will be applied here? We have much stricter information privacy rules in Europe, preventing much of the data collation that microtargeting relies on. But as one senior Irish political organiser put it to me, we already have a highly accurate and granular understanding of the electorate in this country, and it comes from the tallies from every ballot box in every election – combined with the electoral register and the record of who has voted, these tallies give an invaluable map of the electorate’s voting habits, one that every party in the democratic world “would kill to have”, as the organiser put it. Furthermore, the nature of proportional representation means that Irish parties have to appeal to a much broader spectrum of the electorate in order to attain enough preferences, which renders a lot of the advantages of microtargeting redundant.

But undoubtedly some elements of this approach will catch on here. As Issenberg describes it, the benefits are just too significant to ignore. “These all affect races at the margins – they won’t get people to vote for candidates they don’t like, or think the economy is good if it’s bad or make them change their impressions of a party. But they are things that can make campaigns a lot more efficient about knowing where they should devote their resources, whom they should be trying to persuade and whom they should be trying to mobilise, and have given them a much better understanding in behavioural terms about how you motivate people to vote…All these experiments are things that can get you a boost of one point or two points or three points, but if this race is that close, then those interventions add up.”

It remains to be seen if Obama’s lead in data-driven campaigning will make the difference on November 6th, but his role in transforming the fundamental approach to running an election campaign might end up being his most surprising legacy.

Microsoft have just announced their first ever quarterly loss in their corporate history, losing $492 million, albeit with a $6.2 billion write-down after its ill-fated (read: stupid) purchase of online advertising business aQuantive. That write-down is enough to get an asterisk beside this loss, although Microsoft’s ludicrously misjudged Online Services strategy meant something like this was only a matter of time – the division has been losing hundreds of millions every quarter for years. MG Siegler explains why the write-down and subsequent loss is not actually so misleading here; Siegler makes a compelling argument that ”what we’re seeing in Microsoft’s numbers right now is the full-on shift of the company towards enterprise”.

Personally, I have an abiding disdain for most Microsoft products – I had to use XP for years at work and find it insufferable, while the horrendous UX design of Word is borderline abusive. But what I think is most interesting is that the sloppiness and lack of regard for the user one finds in their products is mirrored in their corporate behaviour, a boorish and arrogant demeanour that is hardly conducive to creating carefully considered software. The Vanity Fair article mentioned below details the management “stack-ranking” employee assessment system and how it disincentivises staff, and it all fits – the sort of company that thinks Office’s Ribbon is a positive innovation in user-design is exactly the sort of company that imposes ridiculous managerial bullshit guaranteed to strangle innovation and squash morale. I am just relieved that after years of successfully foisting their junky software on the world, their undeserved hegemony is finally coming to a close.

Which is all a way of introducing my article from last week on Microsoft’s lazy reliance on conventional wisdom, and the high price it is paying for it.

From The Irish Times, Monday, July 9th, 2012

MICROSOFT’S ‘LOST DECADE’ OF RELYING ON CONVENTIONAL WISDOM

Conventional wisdom in the technology industry, as with many fields, builds up over years and even decades of precedent and experience, until it becomes so ingrained that it’s almost impossible to question without looking kind of foolish. Following the conventional wisdom starts off being lucrative, and after a while it becomes a truth that is held to be self-evident and apparently incontrovertible.

But this is a turbulent time in the technology game as we move into the post-PC era, and those self-evident truths that went unquestioned for so long are beginning to look foolish in their own right. The edifice of conventional wisdom is crumbling fast, and it’s taking not just reputations but entire businesses down with it.

One of the oldest and most sacred of those pillars was the belief that Microsoft’s business model was essentially unbeatable. Bill Gates pioneered the horizontally integrated computer business, which saw the production of computers by various original equipment manufacturers, or OEMs, with license fees going to Microsoft for its Windows operating system and ubiquitous Office productivity suite. The margins in this sort of software distribution model were enormous, of course, and Microsoft ended up accruing the lion’s share of profits from the PC industry, while the manufacturers were struggling over ever-decreasing revenues as the whole business became commoditised.

The lesson that everybody took from this, of course, was that horizontal integration was the only viable approach in the computing game. Going any other route, as Apple did under Steve Jobs, was a form of corporate madness that would forever condemn your product to measly marketshare and tiny revenues. The conventional wisdom seemed unshakeable, and plenty of companies thrived by following it to the letter; plenty of analysts, meanwhile, made lucrative careers by ridiculing Jobs’s ideological zeal in retaining full control of the Mac experience.

But that pillar was left in a dusty heap with the news last month that Microsoft was going into the hardware business itself to produce its Surface tablet computer running Windows 8 – this was a seismic change of approach from the Redmond giants, and was an admission that the rules of the game they practically invented have changed dramatically in the past few years. In the post-PC era, the old tactics are no longer winning.

At the launch, Microsoft chief executive Steve Ballmer went out of his way to say how confident he was in his hardware partners, despite the fact that they were now competing with those very same hardware partners – the cognitive dissonance involved in this doublespeak was astonishing. After all, these were the hardware partners who were making dud tablet computers running Microsoft software for a decade, and now that Apple was running away with the category with the iPad, Ballmer was forced to take matters into his own hands.

The ironies here were many and unavoidable – Ballmer was the man who famously, bombastically, laughed at the iPhone before it launched five years ago, saying of Microsoft’s horizontal smartphone approach: “I like our strategy, I like it a lot.” And where once pundits used to pour scorn on Steve Jobs for adhering to a vertically integrated approach with the Mac, now the industry watchers wasted no time in pointing out that Microsoft was not only copying Apple’s products, but they were copying Apple’s entire business model while they were at it.

The criticism of Ballmer’s leadership has been growing for years now, with Microsoft squandering their lead and failing to keep pace with mobile innovation. The most damning judgment of all came in a feature by Kurt Eichenwald in this month’s Vanity Fair magazine that described Ballmer’s “astonishingly foolish management decisions” that lead to a “lost decade”, a chief executive capable only of imposing excessive bureaucracy and managerialism rather than fostering meaningful innovation at Redmond.

After so many years playing catch-up, Ballmer has had to throw out the traditional Microsoft playbook and make drastic changes. But it’s not just Microsoft that is eyeing up a vertical approach – when Google made a mad dash for Motorola last year, everyone assumed Android was going to be benefitting from a more integrated design process from now on, no matter how “open” it stays.

Which brings us to another, related pillar of conventional wisdom that is beginning to look less than secure – the notion that “open” inevitably beats “closed”. Admittedly, these terms were always sort of nebulous in practice and notoriously difficult to pin down, but it’s becoming obvious that there are inherent benefits and weaknesses in both approaches – Google’s difficulties in retaining control over, and deriving profits from, its Android platform is evidence enough of that.

But there is no hard-and-fast rule here – both RIM and Nokia tried to stay vertical and closed, and have keeled over in dramatic fashion, while Samsung is achieving amazing growth while relying on open Android. So is the future vertical and closed, horizontal and open, or kind of diagonal and permeable?

Ultimately, I suspect the only lesson to be taken from all this turbulence is that relying on the conventional wisdom, in any field or business, is a lazy and ultimately risky approach. Instead, it should be clear that challenging the prevailing wisdoms and ignoring the status quo is what real innovation relies on.