Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Why continue to teach and use hypothesis testing (with all its difficult concepts and which are among the most statistical sins) for problems where there is an interval estimator (confidence, bootstrap, credibility or whatever)? What is the best explanation (if any) to be given to students? only tradition? The views will be very welcome.

Thanks in advance

share|improve this question
Duplicate? – csgillespie Feb 7 '11 at 18:09
@csgillespie, seconded, although this might be one of those good duplicates. – mpiktas Feb 7 '11 at 18:21
@mpiktas Yes I wasn't sure. We'll see if anyone votes to close. – csgillespie Feb 7 '11 at 20:47
These quotes are very appropriate. All models are wrong, but some are useful. – mpiktas Feb 8 '11 at 9:15

This is my personal opinion, so I'm not sure it properly qualifies as an answer.

Why should we teach hypothesis testing?

One very big reason, in short, is that, in all likelihood, in the time it takes you to read this sentence, hundreds, if not thousands (or millions) of hypothesis tests have been conducted within a 10ft radius of where you sit.

Your cell phone is definitely using a likelihood ratio test to decide whether or not it is within range of a base station. Your laptop's WiFi hardware is doing the same in communicating with your router.

The microwave you used to auto-reheat that two-day old piece of pizza used a hypothesis test to decide when your pizza was hot enough.

Your car's traction control system kicked in when you gave it too much gas on an icy road, or the tire-pressure warning system let you know that your rear passenger-side tire was abnormally low, and your headlights came on automatically at around 5:19pm as dusk was setting in.

Your iPad is rendering this page in landscape format based on (noisy) accelerometer readings.

Your credit card company shut off your card when "you" purchased a flat-screen TV at a Best Buy in Texas and a $2000 diamond ring at Zales in a Washington-state mall within a couple hours of buying lunch, gas, and a movie near your home in the Pittsburgh suburbs.

The hundreds of thousands of bits that were sent to render this webpage in your browser each individually underwent a hypothesis test to determine whether they were most likely a 0 or a 1 (in addition to some amazing error-correction).

Look to your right just a little bit at those "related" topics.

All of these things "happened" due to hypothesis tests. For many of these things some interval estimate of some parameter could be calculated. But, especially for automated industrial processes, the use and understanding of hypothesis testing is crucial.

On a more theoretical statistical level, the important concept of statistical power arises rather naturally from a decision-theoretic / hypothesis-testing framework. Plus, I believe "even" a pure mathematician can appreciate the beauty and simplicity of the Neyman–Pearson lemma and its proof.

This is not to say that hypothesis testing is taught, or understood, well. By and large, it's not. And, while I would agree that—particularly in the medical sciences—reporting of interval estimates along with effect sizes and notions of practical vs. statistical significance are almost universally preferable to any formal hypothesis test, this does not mean that hypothesis testing and the related concepts are not important and interesting in their own right.

share|improve this answer
Thanks for the interesting list of examples. Given the objective of the question: To contribute to the debate on the review of our statistics courses, we will try to get more details on the implementation of testing in modern devices, can be a great motivation for our engineering students. – Washington S. Silva Feb 8 '11 at 17:28
Most of your examples do not really need classical hypothesis tesing (implyingb a fixed confidence level) but a decision procedure. – kjetil b halvorsen Nov 19 '14 at 15:33
Dear @kjetil: A downvote seems a little harsh here, to be honest. Indeed, the question does not ask anything specific about classical hypothesis testing, and my answer does not make that assumption either! (Hypothesis testing is interpreted broadly here, and with good reason.) – cardinal Nov 19 '14 at 16:06
I need to buy a microwave with auto-reheat . – jmbejara Dec 13 '14 at 8:27
This is a very eloquent answer but I would be very grateful if you explained a little bit more about why all these things are "hypothesis tests". I understand that all your examples are about automated binary decisions. I imagine that in most cases some value is measured and then compared to a cutoff to decide if it is above or below it (and hence arrive to the decision). Does this already qualify as a "hypothesis test" for you, or did you mean something else? I guess when OP asked about why hypothesis testing is still being taught, they did not refer to simple thresholding. – amoeba Feb 5 at 22:11

I teach hypothesis tests for a number of reasons. One is historical, that they'll have to understand a large body of prior research they read and understand the hypothesis testing point of view. A second is that, even in modern times, it's still used by some researchers, often implicitly, when performing other kinds of statistical analyses.

But when I teach it, I teach it in the framework of model building, that these assumptions and estimates are parts of building models. That way it's relatively easy to switch to comparing more complex and theoretically interesting models. Research more often pits theories against one another rather than a theory versus nothing.

The sins of hypothesis testing are not inherent in the math, and proper use of those calculations. Where they primarily lie is in over-reliance and misinterpretation. If a vast majority of naïve researchers exclusively used interval estimation with no recognition of any of the relationships to these things we call hypotheses we might call that a sin.

share|improve this answer
+1, Thanks. Well argued. But in introductory courses, there is no model selection, in the strict sense. You could cite other contexts that are appropriate for the introduction of hypothesis testing? It is acceptable to report the outcome of a test without an estimate of power? – Washington S. Silva Feb 7 '11 at 19:09
Having no model selection in introductory courses isn't a necessity. If you're considering changing a course consider that as a good place to start. – John Feb 7 '11 at 23:42

I personally feel we would be better off without hypothesis tests. The only place I can think of where hypothesis tests offer something unique and useful is in the area of multiple degree of freedom joint hypothesis tests. Examples include ANOVA for comparing more than two groups, simultaneous tests combining main effects and interactions (tests of total effect), and simultaneous tests combining linear and nonlinear terms related to a continuous predictor (multiple d.f. test of association). For simple things, interval estimation is easier, and much less likely to mislead than $P$-values. As said so well in the classic paper Absence of evidence is not evidence of absence, a large $P$-value contains no information. $P$-values only provide evidence against a hypothesis, never in favor of it (Fisher's response when asked how to interpret a large $P$-value was "Get more data"). A confidence or credible interval keeps the researcher more honest by describing how much she doesn't know.

share|improve this answer
I would not that in some fields, "The only place..." and "include ANOVA..." mean you've just covered a tremendous amount of the statistical toolbox. – Fomite Feb 20 '12 at 3:00
I think there's a lot to be said for this position. Given that many researchers mostly want to know about patterns in their data, I've often wondered if we could reasonably set aside much of statistics and simply use plots of the data. (Of course, this assumes the plots would be done skillfully and insightfully, and hypothesis tests wouldn't be as bad if we could say that about them.) – gung Feb 20 '12 at 3:01
Nit-pickingly, I disagree with the quote "absence of evidence is not evidence of absence". Absence of evidence for an effect is not proof that no effect exists, but it certainly constitutes evidence against that effect existing. The question is more about how much evidence against the effect a non-significant result has. The problem with large p-values I think is that in the normal distribution case, large p-values are evidence for the hypothesis, as they are a monotonic function of the goodness of fit. And because the normal distribution is so common, people see this and extrapolate – probabilityislogic Feb 21 '12 at 12:18
Large $P$ means one of many things: the difference is small, the variability too large, or the sample size is too small. Hence the title of the Absence of Evidence paper. – Frank Harrell Feb 21 '12 at 12:53

I think it depends on which hypothesis testing you are talking about. The "classical" hypothesis testing (Neyman-Pearson) is said to be defective because it does not appropriately condition on what actually happened when you did the test. It instead is designed to work "regardless" of what you actually saw in the long run. But failing to condition can lead to misleading results in the individual case. This is simply because the procedure "does not care" about the individual case, on the long run.

Hypothesis testing can be cast in the decision theoretical framework, which I think is a much better way to understand it. You can restate the problem as two decisions:

  1. "I will act as if $H_0$ is true"
  2. "I will act as if $H_\mathrm{A}$ is true"

The decision framework is much easier to understand, because it clearly separates out the concepts of "what will you do?" and "what is the truth?" (via your prior information).

You could even apply "decision theory" (DT) to your question. But in order to stop hypothesis testing, DT says you must have an alternative decision available to you. So the question is: if hypothesis testing is abandoned, what is to take its place? I can't think of an answer to this question. I can only think of alternative ways to do hypothesis testing.

(NOTE: in the context of hypothesis testing, the data, sampling distribution, prior distribution, and loss function are all prior information because they are obtained prior to making the decision.)

share|improve this answer
My goal with the issue was to collect expert opinion in order to enrich the debate on the revision of courses in statistics that is ongoing at the institute where I work in Brazil. The objective is being achieved, with opinions as well placed as of @cardinal, @Andrew Robinson, @probabilityislogic and @JMS. Clearly, hypothesis testing (via N-P, DT or Byes) should be very well taught, but the challenges to build courses as appropriate, given the universality of the teaching of statistics, is equally or more complex than the technique itself. Thank you for your contribution. – Washington S. Silva Feb 8 '11 at 17:19
I love decision theory, if done rigorously using Bayesian methods that incorporate reasonable loss/utility functions. If such functions are not available, I tend to favor interval estimation. – Frank Harrell Feb 20 '12 at 14:48
@FrankHarrell - I agree, but I would still class interval estimation as a kind of "decision theory" where the utility function is usually based on information content (i.e. conclusions which use more of the information we have are better) - and this is optimised by the posterior distribution itself, and possibly a posterior predictive if prediction is of interest. Interval estimation provides a convenient summary of the posterior. And good confidence intervals (e.g. based on MLE) provide a very good approximation to this when the information outside the data at hand is scarce – probabilityislogic Feb 21 '12 at 12:07
usually you use interval estimation when you don't have any specific decision in mind (which is probably the main reason why you wouldn't have a reasonable loss function), and so need to cater for many different scenarios. – probabilityislogic Feb 21 '12 at 12:10

If I were a hardcore Frequentist I would remind you that confidence intervals are quite regularly just inverted hypothesis tests, i.e. when the 95% interval is simply another way of describing all the points that a test involving your data wouldn't reject at the .05 level. In these situations a preference for one over the other is question of exposition rather than method.

Now, exposition is important of course, but I think that would be a pretty good argument. It's neat and clarifying to explain the two approaches as restatements of the same inference from different points of view. (The fact that not all interval estimators are inverted tests is then an inelegant but not particularly awkward fact, pedagogically speaking).

Much more serious implications come from the decision to condition on the observations, as pointed out above. However, even in retreat the Frequentist could always observe that there are plenty of situations (perhaps not a majority) where conditioning on the observations would be unwise or unilluminating. For those, the HT/CI setup is (not 'are') exactly what is wanted, and should be taught as such.

share|improve this answer
Formally speaking, any hypothesis test with alpha bound on the rate of Type I error can be turned into a confidence interval with coverage parameter (1-alpha) and vice versa, no? I don't think you have to be a hardcore frequentist to believe that this is entailed by the definitions. :-) – Keith Winstein Feb 9 '11 at 0:08
@Keith No argument over the definitions, but you do have to be a Frequentist to consider them to be more than interesting and perhaps handy bits of mathematics. That is, if you think sampling theoretic properties are vital for statistical inference then you will (or should) be equally keen on confidence intervals and hypothesis tests since, as we agree, they have this symmetry. Mine was a response to the questioners contrast between 'good' CIs and 'bad' HTs. By lumping them together I wanted to refocus on the contrasts brought up in other answers. – conjugateprior Feb 9 '11 at 8:29

In teaching Neyman Pearson hypothesis testing to early statistics students, I have often tried to locate it in its original setting: that of making decisions. Then the infrastructure of type 1 and type 2 errors all makes sense, as does the idea that you might accept the null hypothesis.

We have to make a decision, we think that the outcome of our decision can be improved by knowledge of a parameter, we only have an estimate of that parameter. We still have to make a decision. Then what is the best decision to make in the context of having an estimate of the parameter?

It seems to me that in its original setting (making decisions in the face of uncertainty) the NP hypothesis test makes perfect sense. See e.g. N & P 1933, particularly p. 291.

Neyman and Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character (1933) vol. 231 pp. 289-337

share|improve this answer
Good tip, thanks. – Washington S. Silva Feb 8 '11 at 17:22

Hypothesis testing is a useful way to frame a lot of questions: is the effect of a treatment zero or nonzero? The ability between statements such as these and a statistical model or procedure (including the construction of an interval estimator) is important for practitioners I think.

It also bears mentioning that a confidence interval (in the traditional sense) isn't inherently any less "sin-prone" than hypothesis testing - how many intro stats students know the real definition of a confidence interval?

Perhaps the problem isn't hypothesis testing or interval estimation as it is the classical versions of the same; the Bayesian formulation avoids these quite nicely.

share|improve this answer
@JMS, "how many intro stats students know the real definition of a confidence interval?" Or, PhD stat graduates, for that matter. – cardinal Feb 8 '11 at 5:11
Quite! Incidentally, I meant no dig at students or practitioners of any stripes. But it's a little crazy to expect the mental gymnastics from someone who didn't sign up for advanced work in statistics. – JMS Feb 8 '11 at 5:32
How many people can say the real definition of CIs? And how many people use them consistently with this definition? Its just too hard not to think "the parameter is likely to be in said interval" - even if you know its not what a CI is. – probabilityislogic Feb 8 '11 at 7:51
E sobre a prática usual de não reportar-se estimativas do – Washington S. Silva Feb 8 '11 at 17:30
What I tried to express is that hypothesis tests not accompanied by estimates of power are very questionable and that interval estimates do not have this additional source of complications. – Washington S. Silva Feb 9 '11 at 12:33

The reason is decision making. In most decision making you either do it or not. You may keep looking at intervals all day long, in the end there's a moment where you decide to do it or not.

Hypothesis testing fits nicely into this simple reality of YES/NO.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.