Interrogating Statistics

Always check statistics. Look at the units, methods, meanings, and source or sources. It’s easy to say “40% of Americans believe that X”–but what does that mean? How was the number derived? Who derived it?

If properly researched, statistics can produce some of the most solid pieces of evidentiary ground in debates. Rigorous investigation should yield scientific levels of accuracy; that is to say, statistics should be able to meet standards of evidence in the hard sciences, namely repeatability (if you try this ten times you should get the same result). That’s pretty solid ground, especially in the frequently mushy fields of social sciences and humanities.

Here’s an example that I saw yesterday on The Economist website. Watch the video if you have a few minutes: it looks fascinating and groundbreaking. Lots of scientific research is wrong!

Let’s hold on for a moment, though. What are the sources for that information? The narrator pulls a series of numbers out of thin air to change the color of the little squares. Without citing a single source. The difference between 20% false positives and 80% is rather significant…and there isn’t any substantive explanation for why the narrator throws those numbers around. Besides which, it’s not as if scientists only test a hypothesis only once. One of the most important rules of science is the repeatability that I mentioned above.

It is in part because statistics are viewed as being solid that it is vital to understand what they actually mean. Quantitative analysis, unlike qualitative analysis, is generally seen as precise and accurate (unless one is predisposed to doubt the source of the information). If you watch that video without looking for citations and methods behind the numbers, you would simply believe the conclusions as they are presented. In a perfect world, we would be able to trust statistics and the people who derive them–but we can’t. The power of numbers is an incitement to manipulation.

What questions, then, should we ask about statistics? The queries vary a lot by discipline–polling and costing estimates are very different–but they can be grouped into three categories.

First, it’s important to know who produced this information. Was it a group of academics, a government agency, the Heritage Foundation, or the Brookings Institution? Just about everyone has an agenda. The most accurate information will come from sources of proven reliability and/or independence (that is to say, a source which has accuracy as its agenda).

Second, you need to find out how the information was produced. This can be very complicated for some types of information, and involve plowing through many pages of figures and calculations. It’s not always terrible, though: good sources will provide some form of abstract or summary and polling agencies usually publish their questions as well as the responses. The wording of polling questions is a great example of how to investigate statistics: slight changes in the phrasing can significantly affect the responses.

Using the first two parts, figure out what the information really means. This sounds simplistic–it isn’t. A number isn’t a number when it has meaning, when it is used in an argument to prove a point. A responsible reader or debater should know what the number means, and just as importantly, what it does not mean. For example, just because Republican favorability ratings are at or below 30% does not mean that Republican candidates will get 30% in House and Senate races next year, or 30% of the popular vote for president in 2016. The number means that about 30% of Americans view the party favorably. The party, not any individual candidate, and only right now. The meaning is derived from the manner in which the information was found (the second group of questions). This is a sort of path-dependency of statistics: if you take a particular route to find an answer, the route will affect the answer.

The key point of this all is to encourage curiosity and examination. Don’t take statistics without thinking about their source, method, and meanings. It’s easy to throw numbers around, and sophists have been doing this for many years. It’s harder, but vital, that we look into the figures that they spout and try to find if those numbers mean what those sophists claim.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s