# What to say to a client that thinks confidence intervals are too wide to be useful?

Suppose I am a consultant and I want to explain to my client the usefulness of confidence interval. The client says to me that my intervals are too wide to be useful and he would prefer to use ones half as wide.

How should I respond?

-
bill them to collect more data. – shabbychef Jun 24 '12 at 4:55
This reminds me of a must-read- paper about types of clients in statistical consultancy. – user10525 Jun 24 '12 at 8:47
@Procrastinator Would you mind posting a link to a PDF version of the paper? – Max Jun 26 '12 at 0:55
@Max, it seems only to be obviously available through JSTOR; posting a PDF would be a violation of JSTOR's (quite reasonable) terms of service ... – Ben Bolker Jun 26 '12 at 9:18
@shabbychef - often this is a highly inefficient way to increase accuracy, especially if the sample is already large. for example, to halve the width of the ci of a normal mean ci you need to quadruple the sample size. better to spend some time improving the model before you go and increase you largest cost fourfold! – probabilityislogic Jul 5 '12 at 9:53

It depends on what the client means by "useful". Your client's suggestion that you arbitrarily narrow the intervals seems to reflect a misunderstanding that, by narrowing the intervals you've somehow magically decreased the margin of error. Assuming the data set has already been collected and is fixed (if this isn't the case, @shabbychef's joke in the comments gives you your answer), any response to your client should emphasize and describe why there's no "free lunch" and that you are sacrificing something by narrowing the intervals.

Specifically, since the data set is fixed, the only way you can reduce the width of the confidence interval is by decreasing the confidence level. Therefore, you have the choice between a wider interval that you're more confident contains the true parameter value or a narrower interval that you're less confident about. That is, wider confidence intervals are more conservative. Of course, you can never just optimize either width or confidence level mindlessly, since you can vacuously generate a $100 \%$ confidence interval by letting it span the entire parameter space and can get an infinitely narrow confidence interval, although it will have $0 \%$ coverage.

Whether or not a less conservative interval is more useful clearly depends both on the context and how the width of the interval varies as a function of the confidence level, but I'm having trouble envisioning an application where using a much lower confidence level to obtain narrower intervals would be preferable. Also, it's worth pointing out that the $95 \%$ confidence interval has become so ubiquitous that it will be hard to justify why you're, for example, using a $60\%$ confidence interval.

-
You can't simply shrink the confidence interval without giving something up, but there is a little flexibility analogous to the difference between a one-tailed test and a two-tailed test. In addition, it is possible that a better model of the same data would produce different (and possibly smaller) confidence intervals. – Douglas Zare Jun 24 '12 at 11:56
I think you're selling the last part just a little bit short. The ubiquity of the 95 % CI is a cultural phenomenon. In some contexts other values are common, eg when plotting, error bars are often standard errors (ie 68% CI's), as I'm sure you're familiar. (+1, btw) – gung Jun 24 '12 at 20:21
In spirit this reply is good but I think the second paragraph is too limiting. One huge benefit the statistician brings to this party is knowledge of alternative procedures that can better meet the client's needs. In many cases one can narrow the width of a CI by selecting a different CI procedure. This neither requires collecting more data (-1 to @shabbychef, I'm afraid) nor decreasing the confidence level. The really hard part is interpreting a CI where the procedure was selected post hoc. That's why we want to have this conversation before analyzing (or even collecting) the data! – whuber Jun 25 '12 at 13:17
I'm reacting to this, Macro, because in practice it does not work well to be so inflexible. The risk is that the client will ignore your advice and turn for help from someone who does not know any better (but claims sufficient expertise in statistics). The OP's scenario is a familiar and common one: it's best seen as an opportunity to inform and educate the client as well as offer them alternatives (along with a frank discussion of their pros and cons). We need to say "yes, the CI can be made smaller, but here are some of the consequences of doing that" rather than "no, you're screwed." – whuber Jun 25 '12 at 13:26
That's a good point @whuber (+1) in situations where more efficient alternatives may be available - yet another reason to consult with a statistician before collecting/analyzing the data. – Macro Jun 25 '12 at 13:40

I would suggest it entirely depends on what your client wants to use the confidence intervals for.

1. Some sort of report/publication/etc. where 95% CI's are normally reported. I might very well tell him "That's not statistically justified" and leave it there, depending on whether or not the client tends to defer to your expertise. If they don't, you have to make a judgement about your own professional comfort with what they want.
2. Some sort of internal document - I'd make it clear you disagree, and make it clear what type of confidence interval the reader is now looking at, since it's not 95%.
3. As a measure of estimate uncertainty, say to determine how much sensitivity analysis one might have to do? I'd give them a figure showing the full distribution with both the 95% CI and something like a 68% CI marked and let them have at it.

I would be quite proud of myself if I managed to keep "So run a bigger study" from being the first thing out of my mouth.

-
+1. I think the comments you made under (2) would probably be relevant in the situation described by (1) as well. – Macro Jun 25 '12 at 12:23

Use the Standard Deviation, as most people do. 95% CI can be scary when people are used to the 68%CI.

-
It sounds to me that in this case we are merely interested in showing the accuracy of, say, the sample mean, not the variability of individual values. Why would you recommend standard deviation, specifically? – chl Jun 24 '12 at 8:51
Fisher originally suggested 95% CIs as an approximation to 2 standard deviations. – Patrick Caldon Jun 25 '12 at 6:27
@Patrick, it sounds like either you missed chl's point (as well as mis-represented Fisher, who made no such mistake) or else you wrote "standard deviation" where you intended "standard error". Most CIs are based on standard errors, of course, not standard deviations. 2 SDs neither approximates a CI nor vice versa. – whuber Jun 25 '12 at 13:21
Of course, the standard error is just the standard deviation of the mean, so it's just terminology. That is, saying that CIs aren't based on standard deviations isn't really true. They're not based on the standard deviation of the sample, but instead on the standard deviation of the mean. – Aaron Jun 25 '12 at 20:00
Not all estimates are means. There are standard errors for estimates other than means and it is the standard error of the estimate that is used to generate confidence intervals for a parameter based on the variability of an estimate as whuber suggests. – Michael Chernick Jun 25 '12 at 22:56

You provide a confidence interval at a certain standard level such as 90% or 95%. The client can judge whether or not the interval is too wide to be useful. But of course that does not mean that you can shorten it to make it useful. You can suggest that increasing the sample size will decrease the width of an interval at a given confidence level as it decreases roughly by a factor of the square root of the sample size.

-