November/December 2000 | Contents

POLLS SPEED DOWN SLIPPERY SLOPE, BUT THEY DON'T HAVE TO

BY ANDREW KOHUT

Another election cycle has concluded, and I'm sure the polls will give a good accounting of why the election turned out as it did. But I've been troubled by the way polls were used in this election period. The pollsters and the news media have both contributed to the problems.

Polling is being used by the press less as a check on the conventional wisdom about voter opinion, and more as a way of underscoring the running story line with numbers. Polls are often conducted at the worst possible times for getting a reasoned reaction from voters to campaign events. They are reported breathlessly and then denounced when their ephemeral effects disappear, or conflict with what other polls subsequently show. Polls, and some pollsters, have become part of the tabloidization of political reporting. And more and more polls are being conducted on a judgmental rather than on a systematic basis.

These are not new problems. Polling has been on a slippery slope for some time, its credibility increasingly challenged. It does not have to be so. There are many ways in which the new information environment can enhance rather than worsen public opinion research.

PRESS PROBLEMS

Poll results too often are used to support an election news narrative and/or annotate the conventional wisdom. Using poll results that fit a story line is not new, but there are so many polls now that it's easy to find one that lends authority to a reporter's or editor's notions.

The attention given the results of a Zogby poll of Republican voters in South Carolina -- taken one day after Senator John McCain's victory in New Hampshire and showing the Arizona Senator with a 44 percent to 39 percent lead -- is the best case of this in campaign 2000. Based on one night of interviewing (February 2), it supported, if not created, the idea that McCain could triumph in that very loyalist GOP state. Zogby's findings were used widely and prominently by news organizations, many of whom are well aware of the tenuousness of survey results conducted so quickly on the heels of an important primary. A Lexis/Nexis search found the Zogby poll making forty-two newspapers -- fourteen of which played it on page one. Three network TV broadcasts reported it, as did sixteen cable news programs. Was it big news, or was it news that fit the contagion of the media moment? Evidence suggests the latter. McCain would never hold a significant lead in any South Carolina poll except those taken in the immediate aftermath of his big New Hampshire win. In fact, as early as February 6, the ABC News/Washington Post poll had Bush in front by a significant margin, and he won the election by 53 percent to 42 percent.

The timing of polls to catch the bounce of a big event has created an unrealistic impression of voter volatility and makes polls look silly to ordinary people. That's particularly true of polls taken during, and just after, the nominating conventions. This year's prize goes to CNN and USA Today for ordering up a one-night Gallup poll after Al Gore's choice of Senator Joe Lieberman as his running mate. It found Bush's 19-percentage-point lead evaporating instantly, only to reappear a few days later. Similarly, Newsweek and others that polled just after each of the conventions reported dizzying shifts in opinion this year, as the unstable opinions of voters with low conviction were buffeted by news events.

These polls seem almost calculated to portray instability in public opinion rather than look at the underlying factors that make many voters unsure of their choice. The bounce is real, but often short-lived, and always confusing to audiences. People throw their hands up and dismiss the mystifying patterns in the polls, with good reason.

POLLSTER PROBLEMS

Differences in survey methods also added to polling confusion this year. There are legitimate debates about when and how to distinguish between eligible voters and those most likely to vote. But some polls are now using what poll takers call "judgment based" samples. Since the 1950's, the major polls -- Gallup, Harris, and the leading national news organizations -- have used scientifically random methods to draw their samples. But polls produce imperfect surveys because they must be conducted in short periods of time, often under-representing difficult-to-reach and uncooperative respondents. The samples are routinely corrected using demographic census data on sex, age, race, education, and other factors to help overcome sampling imbalances.

Newer polls, however -- mostly those conducted by campaign consultants -- take a different approach. They make corrections based on the pollster's suppositions about voters' party affiliation -- the percent who think of themselves as Republicans, Democrats, or independents. The problem is that party affiliation is an attitude, not a stable characteristic such as age or educational attainment. Furthermore, self-identification with the political parties changes over the course of campaigns, especially when one candidate has a big lead.

There has always been plenty of room for different approaches in polling, but personal judgments about the composition of the sample have not been part of the equation. This represents a real step backward in survey methodology and would have been unthinkable twenty years ago when pollsters prided themselves on systematic rather than idiosyncratic methods. It can also make a real difference. The Voter.com Battleground 2000 poll employing this method was the only survey to have Bush ahead around Labor Day when Gore was surging. Pundits, and the public alike, wondered why.

Pollsters have also become part of a tabloid approach to political reporting. Cable news programs often interview small samples of people registering their instant reactions to debates and speeches. Those interviews do not represent public opinion, only impromptu emotional responses.

MSNBC's regular use of the partisan GOP pollster Frank Luntz, without a Democratic counterpart, caught the attention of the Polling Review Board of the National Council on Public Polls, an association of pollsters who work for the news media. The board protested the partiality, noting that Luntz is deeply involved in Republican Party affairs, and once was cited by the American Association for Public Opinion research as being in violation of its ethics code.

While there is still a lot of good polling in news media reports, its value is diminished by these trends. As journalism has begun to rethink its values and practices, pollsters and the news organizations that employ them need to think through some of the ways that polling adds to the clutter and confusion in political reporting. The Internet, the twenty-four-hour news cycle, and an unquenchable desire for information all make this a potentially great time for polling. The issue is how to meet those needs with surveys that advance the principles of good journalism and serve the public interest.

Andrew Kohut, director of the Pew Research Center for the People and the Press, writes regularly for CJR about public attitudes toward the media.