More Damned Lies and Statistics

How Numbers Confuse Public Issues

Joel Best

Lunch was at a prominent conservative think tank. The people around the table were fairly well known; I’d read some of their books and articles and had even seen them interviewed on television. They listened to me talk about bad statistics, and they agreed that the problem was serious. They had only one major criticism: I’d missed the role of ideology. Bad statistics, they assured me, were almost always promoted by liberals.

Two months earlier, I’d been interviewed by a liberal radio talk-show host (they do exist!). He, too, thought it was high time to expose bad statistics—especially those so often circulated by conservatives.

When I talk to people about statistics, I find that they usually are quite willing to criticize dubious statistics—as long as the numbers come from people with whom they disagree. Political conservatives are convinced that the statistics presented by liberals are deeply flawed, just as liberals are eager to denounce conservatives’ shaky figures. When conservatives (or liberals) ask me how to spot bad statistics, I suspect that they’d like me to say, "Watch out for numbers promoted by people with whom you disagree." Everyone seems to insist that the other guy’s figures are lousy (but mine are, of course, just fine, or at least good enough). People like examples of an opponent’s bad statistics, but they don’t care to have their own numbers criticized because, they worry, people might get the wrong idea: criticizing my statistics might lead someone to question my larger argument, so let’s focus on the other guy’s errors and downplay mine.

Alas, I don’t believe that any particular group, faction, or ideology holds a monopoly on poor statistical reasoning. In fact, in choosing examples to illustrate this book’s chapters, I’ve tried to identify a broad range of offenders. My goal is not to convince you that those other guys can’t be trusted (after all, you probably already believe that). Rather, I want you to come away from this book with a sense that all numbers—theirs and yours—need to be handled with care.

This is tricky, because we tend to assume that statistics are facts, little nuggets of truth that we uncover, much as rock collectors find stones.¹ After all, we think, a statistic is a number, and numbers seem to be solid, factual proof that someone must have actually counted something. But that’s the point: people count. For every number we encounter, some person had to do the counting. Instead of imagining that statistics are like rocks, we’d do better to think of them as jewels. Gemstones may be found in nature, but people have to create jewels. Jewels must be selected, cut, polished, and placed in settings to be viewed from particular angles. In much the same way, people create statistics: they choose what to count, how to go about counting, which of the resulting numbers they share with others, and which words they use to describe and interpret those figures. Numbers do not exist independent of people; understanding numbers requires knowing who counted what, why they bothered counting, and how they went about it.

All statistics are products of social activity, the process sociologists call social construction. Although this point might seem painfully obvious, it tends to be forgotten or ignored when we think about—and particularly when we teach—statistics. We usually envision statistics as a branch of mathematics, a view reinforced by high school and college statistics courses, which begin by introducing probability theory as a foundation for statistical thinking, a foundation on which is assembled a structure of increasingly sophisticated statistical measures. Students are taught the underlying logic of each measure, the formula used to compute the measure, the software commands used to extract it from the computer, and some guidelines for interpreting the numbers that result from these computations. These are complicated lessons: few students have an intuitive grasp of any but the simplest statistics, and instruction usually focuses on clarifying the computational complexities.

The result is that statistical instruction tends to downplay consideration of how real-life statistics come into being. Yet all statistics are products of people’s choices and compromises, which inevitably shape, limit, and distort the outcome. Statistics instructors often dismiss this as melodramatic irrelevance. Just as the conservatives at the think tank lunch imagined that bad statistics were the work of devious liberals, statistics instructors might briefly caution that calculations or presentations of statistical results may be "biased" (that is, intentionally designed to deceive). Similarly, a surprisingly large number of book titles draw a distinction between statistics and lies: How to Lie with Statistics (also, How to Lie with Charts, How to Lie with Maps, and so on); How to Tell the Liars from the Statisticians; How Numbers Lie; even (ahem) my own Damned Lies and Statistics.² One might conclude that statistics are pure, unless they unfortunately become contaminated by the bad motives of dishonest people.

Perhaps it is necessary to set aside the real world in an effort to teach students about advanced statistical reasoning. But dismissive warnings to watch out for bias don’t go very far in preparing people to think critically about the numbers they read in newspaper stories or hear from television commentators. Statistics play important roles in real-world debates about social problems and social policies; numbers become key bits of evidence used to challenge opponents’ claims and to promote one’s own views. Because people do knowingly present distorted or even false figures, we cannot dismiss bias as nonexistent. But neither can we simply categorize numbers as either true figures presented by sincere, well-meaning people (who, naturally, agree with us) or false statistics knowingly promoted by devious folks (who are on the other side, of course).

Misplaced enthusiasm is probably at least as common as deliberate bias in explaining why people spread bad statistics. Numbers rarely come first. People do not begin by carefully creating some bit of statistical information and then deduce what they ought to think. Much more often, they start with their own interests or concerns, which lead them to run across, or perhaps actively uncover, relevant statistical information. When these figures support what people already believe—or hope, or fear—to be true, it is very easy for them to adopt the numbers, to overlook or minimize their limitations, to find the figures first arresting, then compelling, and finally authoritative. People soon begin sharing these now important numbers with others and become outraged if their statistics are questioned. One need not intentionally lie to others, or even to oneself. One need only let down one’s critical guard when encountering a number that seems appealing, and momentum can do the rest.

The solution is to maintain critical standards when thinking about statistics. Some people are adept at this, as long as they are examining their opponents’ figures. It is much more difficult to maintain a critical stance toward our own numbers. After all, our numbers support what we believe to be true. Whatever minor flaws they might have surely must be unimportant. At least, that’s what we tell ourselves when we justify having a double standard for judging our own statistics and those of others.

This book promotes what we might call a single standard for statistical criticism. It argues that we must recognize that all numbers are social products and that we cannot understand a statistic unless we know something about the process by which it came into being. It further argues that all statistics are imperfect and that we need to recognize and acknowledge their flaws and limitations. All this is true regardless of whether we agree or disagree with the people presenting the numbers. We need to think critically about both the other guys’ figures and our own.

I should confess that, in writing this book, I have done little original research. I have borrowed most of my examples from works by other analysts, mostly social scientists and journalists. My goal in writing about bad statistics is to show how these numbers emerge and spread. Just as I do not believe that this is the work of one political faction, I do not mean to suggest that all the blame can be laid at the door of one segment of society, such as the media. The media often circulate bad numbers, but then so do activists, corporations, officials, and even scientists—in fact, those folks usually are the sources for the statistics that appear in the media. And, we should remember, the problems with bad statistics often come to light through the critical efforts of probing journalists or scientists who think the numbers through, discover their flaws, and bring those flaws to public attention. A glance at my sources will reveal that critical thinking, just like bad statistics, can be found in many places.

The chapters in this book explore some common problems in thinking about social statistics. The chapter titles refer to different sorts of numbers—missing numbers, confusing numbers, and so on. As I use them, these terms have no formal mathematical meanings; they are simply headings for organizing the discussion. Thus, chapter 1 addresses what I call missing numbers, that is, statistics that might be relevant to debates over social issues but that somehow don’t emerge during those discussions. It identifies several types of missing numbers and seeks to account for their absence. Chapter 2 considers confusing numbers, basic problems that bedevil our understanding of many simple statistics and graphs. Scary numbers—statistics about risks and other threats—are the focus of chapter 3.

The next three chapters explore the relationship between authority and statistics. Chapter 4’s subject is authoritative numbers. This chapter considers what we might think of as statistics that seem good enough to be beyond dispute—products of scientific research or government data collection, for instance. It argues that even the best statistics need to be handled with care, that even data gathered by experts can be subject to misinterpretation. Chapter 5 examines what I call magical numbers—efforts to resolve issues through statistics, as though figures are a way to distill reality into pure, incontrovertible facts. Chapter 6 concentrates on contentious numbers, cases of data duels and stat wars in which opponents hurl contradictory figures at one another. Finally, chapter 7 explores the prospects for teaching statistical literacy, for improving public understanding of numbers and teaching people how to be more thoughtful and more critical consumers of statistics.

The lesson that people count—that we don’t just find statistics but that we create them—offers both a warning and a promise. The warning is that we must be wary, that unless we approach statistics with a critical attitude, we run the risk of badly misunderstanding the world around us. But there is also a promise: that we need not be at the mercy of numbers, that we can learn to think critically about them, and that we can come to appreciate both their strengths and their flaws.