I've been reading a number of articles and blog posts about issues relating to hypothesis testing. While these sources seem to put forth legit problems with hypothesis testing and it's interpretations, I'm not seeing anything in terms of alternatives to it (beyond bayesian hypothesis testing). What are some alternative to hypothesis testing in R?

Problems with the hypothesis testing approach

So-called Bayesian hypothesis testing is just as bad as regular hypothesis testing

Alternatives to Statistical Hypothesis Testing


For example, let's consider the following situation. We have monthly data for one year where conversions_a is the control and conversions_b is the experimental data for conversion when using a different headline.

df = data.frame(year_one_a = 1:12, conversions_a = rnorm(12), 
                year_one_b = 1:12, conversions_b = rnorm(12)+5)
  • 3
    $\begingroup$ In many cases, intervals (which show effect sizes) are more meaningful that hypothesis tests. $\endgroup$ – Glen_b Feb 1 '14 at 13:10
  • 2
    $\begingroup$ I think the question needs some context to be answerable. As @Glen says, confidence intervals around a parameter estimate are often more useful than a test that the parameter's true value is exactly zero, but what kind of situations are you thinking of? $\endgroup$ – Scortchi - Reinstate Monica Feb 1 '14 at 13:20
  • $\begingroup$ I wasn't thinking of a specific situation, just looking to mimic the following question but for hypothesis testing. stats.stackexchange.com/questions/2234/… $\endgroup$ – amathew Feb 1 '14 at 13:26
  • 1
    $\begingroup$ I think it depends on the kind of experiment you're doing, i.e. observational v/s experimental, and the kind of questions you're asking. Is there any kind of particular setting you're interested in? $\endgroup$ – Stijn Feb 1 '14 at 13:29
  • $\begingroup$ possible duplicate of Effect size as the hypothesis for significance testing $\endgroup$ – gung - Reinstate Monica Feb 1 '14 at 14:33

One alternative is to forgo p-values altogether and focus on what the results mean. In many situations, p-values answer a question we are not (or ought not be) interested in:

If, in the population from which this sample was drawn, there is no effect, how likely is it that, in a sample of this size, we would get a test statistic as big or bigger than the one we got?

Instead, we ought to be interested in what the statistics add to an argument about what is going on. This idea is fully developed in the book "Statistics As Principled Argument" by Robert Abelson (link goes to my review) but, essentially, we ought to be asking these questions about the effects we find:

  1. How big are they?
  2. How precise are they?
  3. How widely do they apply?
  4. How interesting are they?
  5. How credible are they?

This can only be done if we are also substantive experts or if we work closely with experts. For example, sometimes small effects are quite interesting - so interesting that they are worth discussing. Indeed, sometimes effects are interesting because they are small - if the literature and theory suggests that they ought to be big.

Highly credible claims require less evidence than ones that are not credible, but credibility has non-statistical sources.

  • 1
    $\begingroup$ +1, this is the best statement of this point of view that I've seen from you on CV. $\endgroup$ – gung - Reinstate Monica Feb 1 '14 at 15:10
  • $\begingroup$ +1 Excellent suggestions! Of course, one need not forgo p-values in order to consider the answers to your questions. Instead one has to forgo the silly binary determination of 'significant' versus 'not significant'. $\endgroup$ – Michael Lew Feb 1 '14 at 21:14
  • $\begingroup$ Attempts to face this issue without making clear the distinction between hypothesis tests and significance tests lead to throwing out the baby with the bathwater (p-values are the baby, hypothesis tests are the bathwater). $\endgroup$ – Michael Lew Feb 1 '14 at 21:21
  • $\begingroup$ Then, @MichaelLew how do you propose testing hypotheses? I'd be eager to read it (would probably need to be a separate answer, not a comment). $\endgroup$ – Peter Flom Feb 1 '14 at 21:25
  • $\begingroup$ When would you ever want a statistical test to dictate your actions in a manner that excludes addressing the types of questions that you list? If you, like I, would answer nearly never then you nearly never need a hypothesis test. For people interested in evidence the p-value from a significance test can be interesting, but the dichotomous result of a hypothesis test is not. I've addressed these issues in an extensive form in this paper: ncbi.nlm.nih.gov/pubmed/22394284. $\endgroup$ – Michael Lew Feb 1 '14 at 23:21

This is subjective, and doesn't directly address your question, but I hope it's helpful regardless.

I think people too often conflate a statistical hypothesis with a scientific one, which leads to problems. Regarding the former, the statistical hypothesis being tested is almost always whether the parameter being estimated is zero or not. But with any deeper thought on the matter, one realizes this is often trivial, for all the reasons Peter stated. But because it's called a hypothesis, and it is falsifiable, it seems to satisfy people that something scientific has been achieved.

It also tends to reduced a complex scientific question to the reporting of one key statistical hypothesis. This is further encouraged by how a lot of scientific research is published - thin-slice the bigger problem and report every slice/p-value in a different paper.

Where I am going is this, with respect to your question? I am not offended by statistical hypothesis tests per se, as long as they are applied thoughtfully. In other words, the solution is not just about finding a better statistical approach, but rather about applying a better scientific approach; by clearly specifying a complete theory that can be tested, collecting the relevant data, and then ensuring any statistical modelling or testing directly follows from these, rather than the other way around.


Following the ideas put forward here http://robjhyndman.com/working-papers/forecasting-without-significance-tests/, you can formulate two models: with and without a dummy variable. Then you can use AIC-like statistics (and start reading about their own problems and misunderstandings).


Not the answer you're looking for? Browse other questions tagged or ask your own question.