Forum

In this new post to the BDI Forum, Tom Wein examines the use of data and evidence in the US presidential election, in the UK and in the media, concluding that this hugely valuable trend still has far to go.

For scale, expense and drama, there is nothing to match a US presidential election. It is surely the biggest behavioural change event in the world. As with all behavioural change events, the first, and one of the trickiest, challenges is to understand the audience. For that reason, the Republican and Democratic parties have made enormous efforts to delve into and understand the electorate.

In the 2000s, political parties began to develop databases to help them identify likely voters and target communications. The Republicans’ Voter Vault and the Democrats’ Datamart both launched in 2002, and soon hit 160 million entries. Yet their numbers and sophistication are dwarfed today: a 2004 New York Times story explained that only “creative imaginations” would believe the rumours that the databases included car makes and dog breeds. With personal information more available online, and computerized analysis techniques far more sophisticated, today’s databases surely could, if they wanted to.

They may not yet be recording dog breed – but that may be the only thing they aren’t. Time Magazine reports that when Jim Messina took over Barack Obama’s re-election campaign, he said: “We are going to measure every single thing in this campaign.” Under the leadership of ‘chief scientist’ Rayid Ghani, the Chicago HQ amalgamated their many databases into a single huge resource. Using this information, they ran controlled trials to determine which campaign figures could raise the most money through email appeals, and what form of words they ought to use. They ranked call lists in order of persuadability, bought media space more effectively, and relentlessly tested their assumptions against the data. In short, they fought to understand their audience.

Republicans have generally been slower. Their pricey great hope – the ORCA app that would track voting in real time – failed dismally on election day, as Politico reports. Sasha Issenberg, author of ‘The Victory Lab: The Secret Science of Winning Campaigns’ spoke to the Washington Post, noting that the Right has found it much tougher to attract the most talented political scientists and economists. That Democratic advantage in data crunching was surely lamented by Mitt Romney, who said to the Wall Street Journal: “I love data…I used to call it ‘wallowing in the data…Let me see the data. I want to see the client’s data, the competitors’ data. I want to see all the data.”

Not just the campaigns

Just as the campaigns have looked for hard data to back up their assumptions, so too have the media – but the move has not been without controversy. For those seeking data, the flagbearer has been Nate Silver. His FiveThirtyEight blog at the New York Times aggregated polling in each state, constantly updating the likelihood of each candidate’s victory. His prediction of an Obama victory led to much strident criticism, not just from GOP supporters, but also from political pundits who favour their experience and contacts over mathematical models. His near-perfect record has led to announcements of ‘the end of punditry’. If that is not quite nigh, it is certainly true that traditional columnists will have to take greater account of the data to understand the electorate from now on.

The military too have looked to move towards data. With near-universal agreement that kill-counts were insufficient to judge progress in Afghanistan, the Pentagon launched the Human Terrain Systems programme. Though it has faced its problems, the basic idea of collecting data to understand the audience was considered central to a ‘hearts and minds’ war. Acting on the same impulse, the military sought to devise innovative metrics, such as the availability of exotic fruits at Afghan marketplaces - a measure of a functioning trade economy. It was entirely appropriate that counter-insurgency guru David Kilcullen’s next challenge after Afghanistan was as CEO of Caerus Associates, applying Big Data analysis to tricky problems in politics, conflict and business.

This side of the pond

The US election is not the only home of the new rise of data. In the UK, parties have slowly developed equivalent voter databases. Prime Minister Cameron is currently trialing an iPad app giving him real time access to economic and polling data. The key figure behind the drive for data is said to be Rohan Silva. A senior policy advisor to Mr Cameron, he speaks simply of using technology to better society.

A centerpiece of that has been government data releases. Beloved of bloggers everywhere, many hoped that these would bring a new accountability, with citizens scouring budget lines. So far, these dreams have not been realised. Much data was not comparable, or released in unfriendly formats, and users complained of jargon without context. Now relaunched with a flashy new website, the government points to a number of small but satisfying innovations, including advice for students and cyclists. Lewisham Council in London reports £20,000 of savings from a scheme to keep streets clean. Ministers hope that bright young things will find more handy ways to use the numbers.

The government is also seeking to increase the use of policy experiments. Much has been written about the Cabinet Office’s “Nudge Unit”, and their application of psychological insights. Yet their lasting legacy may be their reliance on evidence. Randomized Controlled Trials see a policy administered to one group but not another, and the results compared. They are now common in medicine, but much rarer in government. There, evidence-based policy is often hailed but rarely practised, as ministers weigh academic papers against headlines; Gordon Brown’s Home Secretaries struggled to balance scientific advice on drugs with moral messaging. This time, there may be more clout behind the notion. The focus for now is on uncontroversial topics – paying taxes and the like – before expanding experimentation into more ideologically charged arenas. Here again, field research to understand the audience is superseding political hunches and prejudice.

Government advances have once again been paralleled by media changes. Numerate journalists such as Mark Easton and Daniel Finkelstein have received acclaim. Popular authors such as Ben Goldacre and Mark Henderson have highlighted the statistical ignorance of policymakers and journalists – and the way savvy drugs companies and campaign groups exploit that. Henderson’s book, the Geek Manifesto, has even been the subject of a public campaign, with fans sending them to MPs. That such a campaign exists – but that it is necessary shows how much there is still to do.

Much to do

Less welcome for the government has been the criticism of Sir Michael Scholar. The feisty head of the UK Statistics Authority was quick to criticize the many politicians who abuse official statistics. Immigration and crime statistics were particular battlegrounds during his tenure. (He retired in March 2012. His replacement, Andrew Dilnot, is not known for his reticence, but may be more politic in his criticisms).

The relationship between drugs policy and evidence has likewise been deeply controversial. In 2009, the Home Office dismissed Professor David Nutt for his blunt criticism of a government policy that classified drugs according to moral criteria, rather than data on actual harms. That view still holds. In October 2012, the UK Drugs Policy Commission published a major report, the culmination of 6 years of research. Announcing it, the charity’s chairman, Dame Ruth Runciman, said that much of UK drugs policy “does not have an adequate evidence base…We spend billions of pounds every year without being sure of what difference much of it makes.” The report was tersely dismissed by the Home Office – though a spokeswoman thanked them for their work.

The media likewise continue to routinely abuse statistics. The UK Statistics Authority was sharp in its criticism of the Daily Mail’s coverage of the London riots. Though the Daily Mail sin above all others, all the major newspapers are routinely guilty: guilty of seeking the most dramatic gobbet, rather than the accurate figure; or guilty of cherry picking anecdotal evidence to accord with their prejudices.

Where next?

The Behavioural Dynamics Institute believes in evidence. Untested assumptions and ‘black box’ creativity are dangerous enemies of effective behavioural change. Understanding the audience means asking them, and observing them, not second-guessing them.

For now, the first task for those who believe in evidence is to evangelize. There are still large swathes of government and the media – on both sides of the Atlantic – who do not trust evidence when it comes in number format. They must be educated.

Yet there are also important changes to be made in the way audience research is conducted.

First, quantitative research is a limited tool. Sufficient for measuring an already understood environment, it falls down badly abroad, where it cannot make links or explain the unexpected finding. Polling firms such as Gallup have struggled to do more than describe the human terrain in locations such as Afghanistan. Quantitative data collection must be preceded, complemented, and followed by qualitative data collection, to provide a rounded analysis and effective behavioural change recommendations.

Second, quantitative data must be aggregated into models to be truly useful. Yet those models attempted rigour can often disguise the assumptions below. The use of numbers does not make a calculation objective, even if it is a useful start. That lesson was painfully demonstrated by the failure of financial risk models to predict the crash. (A lack of clarity about his model has been a key criticism too of Nate Silver). The assumptions behind models must be made clearer, if they are to be relied on in understanding audiences for such vital tasks as elections and health campaigns.

Third is a point which is particularly close to our hearts at the Behavioural Dynamics Institute. Current evidence attempts are still dealing with externally dictated demographic cohorts, and not self-identifying social groups. Though the Romney campaign sometimes spoke of soccer moms, or blue-collar white men, most analysis focused on Obama’s lead among women, or Romney’s deficit among Hispanics. That led to some shallow advertising, in which women’s issues were reduced to Roe v Wade, and Hispanics seemed to care only about immigration. Approaching people as they live their lives, rather than according to census-derived criteria, will deliver better evidence and better behavioural change campaigns.

The rise of data is a fine thing. Understanding the audience – whether that is voters, or patients, or union members, or enemies – is vital to quality communication and effective policy. The more rigorously it can be done, and the more assumptions that can be tested, the better. And if that means asking people about the ancestry of their pets, then that is what we shall ask.