What Is a Confidence Interval and Why Would You Want One?

Using the Central Limit Theorem to Find a Confidence Interval

If you’re still with me, let’s get back to our challenge: deciding whether our original mean of 54.0 seconds from the first eight participants was sufficiently convincing to show that we’d met our target of an average time on task of less than 60 seconds and would allow us to launch. We’d rather not run nine more rounds of usability tests; instead, we want to estimate the true population mean.

Fortunately, the Central Limit Theorem lets us do that. Any mean from a random sample is likely to be quite close to the true population mean, and a normal distribution models the chance that it might be different from the true population mean. Some values of the true population mean would make it very likely that I’d get this sample mean, while other values would make it very unlikely. The likely values represent the confidence interval, which is the range of values for the true population mean that could plausibly give me my observed value.

To do the calculation, the first thing to decide is what we’re prepared to accept as likely. In other words, how much risk are we willing to run of being wrong? If we’re aiming for a level of risk that is often stated as statistical significance at p < 0.05, the risk is a 5% chance of being wrong, or one in 20, but there is a 95% chance of being right.

The next thing we need is a standard deviation. The only one we have is the standard deviation of our sample, which is 29.40 seconds. (I used Excel’s STDEV.S command to work that out.)

Finally, we plug in the mean, which is 54.0 seconds, and the number of participants, which is 8.

You can work this out with formulas and a calculator, but let’s use Excel. The CONFIDENCE command does it, giving us a value that we can

• subtract from the sample mean to get the lowest true population mean that our observed mean could plausibly have come from
• add to the sample mean to get the highest true population mean that our observed value could plausibly have come from

The result: the 95% confidence interval for the mean is 29.4 to 78.6 seconds, in comparison to our target of 60 seconds.

This is unfortunate. If the true population mean were as high as 78.6 seconds, we could still have obtained our sample mean of 49.4 seconds with a 95% probability. Oh, dear. That would be 18.6 seconds greater than our task-time target, which is disappointing all around. But we wouldn’t be nearly as worried if the true population mean happened to fall at the low end of the range. That would mean we’ve met our target.

Confidence Intervals Aren’t Always Correct

Remember that 95%, which says that about one time in 20 you’re likely to get it wrong? You wouldn’t know whether this time is the one time in 20. If that makes you feel uncomfortable, you’ll need to increase your confidence level, which will also increase the range of the confidence interval, so you’ll have a greater chance of catching the true population value within it.

Here are the confidence intervals for this sample, for some typical levels of risk:

		Confidence Interval (in seconds)
Risk of Being Wrong	Confidence Level	Lower End	Upper End
20%	80%	39.3	68.7
10%	90%	34.3	73.7
5%	95%	29.4	78.6
1%	99%	17.6	90.4

You can see that, as we reduce the risk, we increase the confidence level and end up with a wider confidence interval—and in this example, also have an increasing level of depression about that launch date.

Have you come across Six Sigma, the quality improvement program that Motorola originated, which is now popular in many manufacturing companies? They wanted to be very, very sure that they knew the risk of manufacturing poor-quality products and chose a confidence level of 99.99966%—that is, 3.4 chances in a million. I didn’t bother calculating the confidence interval for our sample to get a Six-Sigma level of risk, because it would constitute the whole range of our data.

Confidence Intervals Depend on Sample Size

What to do if you want to get a higher confidence level, but also need to be sure you’ve met your target for the mean? Increase the sample size.

The more data in your sample, the smaller your confidence interval. That’s because with more data, you have more chance of the sample being a pretty good match to the whole population and, therefore, of its mean being similar to the true population value.

In my example, I’ve got 80 participants overall. The mean for all of the participants is 47.1 seconds, and the 95% confidence interval is (39.8, 54.4). So if I’d tested with a lot more people, I would indeed have proven that we’re okay to launch, because the highest plausible value is less than my target of 60 seconds.

That’s part of the fun of confidence intervals: we want to calculate a confidence interval so we don’t have to do as much sampling, but to get a narrow confidence interval, we need to do more sampling.

The Central Limit Theorem Works Only on Random Samples

I recently read an article on sample sizes that asserted, “One thousand sessions provide a sufficiently narrow margin of error (plus or minus 2.5% at a 90% confidence level).”

This is true, but only if the sample is a random sample. For example, let’s say we wanted the average time it took to complete the New York Marathon across 45,000 runners. If we took a random sample of just 1000 runners, we would get a narrow confidence interval. But if we took the times of the first 1000 runners across the finish line, we’d get something very far indeed from the true population mean.

The Mean Is Convenient, But Not Always Helpful

So far, I’ve used the example of a target for a mean value: The average time on task must be less than a specified target. But would that be a good target to have?

Figure 5 shows a set of data that is quite typical for user experience: a peak at low values—for example, task times—then a long tail with a few values that are much higher. It’s the overall data set that our samples have so far come from. The mean is 50.3 seconds, which is lower than the target of 60 seconds.

But how useful is the mean? Suppose we advertised: On average, you’ll be able to do this task in less than 60 seconds. In fact, some of our participants’ task times are much longer than the target—8% of the data set has values over twice as long, and the largest value is over five times as long. Okay, so that’s only five minutes, and maybe no one would notice. But what if we were working in minutes instead of seconds? Many people would indeed notice if a task that they anticipated taking less than an hour actually took over two hours. So, in user experience, we often need to know the range—and you can’t calculate a confidence interval for that.

Also, look at the way most of the values pile up at the shorter end. Those users ought to be happy—the time it took them to complete the task is much shorter than the advertised time. But to ensure that a high volume of users can achieve those task times once the system gets rolled out, we’ll have to make sure that the system can cope with that high peak of very fast task times. So our colleagues who are managing system performance are likely to be far more interested in the most frequent value, which is the mode, than in the mean. But you can’t calculate a confidence interval for the mode either.

Of course, Excel can take any numbers you put in and shove them through the calculation—so if you mistakenly try to run a CONFIDENCE formula on a mode, you’ll get an output. But it won’t be meaningful, because there is no Central Limit Theorem or any equivalent for modes.

Summary: Confidence Intervals Can Save You Effort

The confidence interval for the mean helps you to estimate the true population mean and lets you avoid the additional effort that gathering a lot of extra data would require. You can compare the confidence interval you calculated with the target you were aiming for.

Once you have worked out what level of risk you are willing to accept, confidence intervals for the mean are easy to calculate. You’ll need these formulas in Excel:

• AVERAGE—to get the mean of your sample
• STDEV.S, in Excel 2010, or STDEV, in earlier versions—to get the standard deviation of your sample
• CONFIDENCE—to calculate the amount to subtract from the mean to get the lower end of the confidence interval and to add to the mean to get the upper end