In his blog posts, Andrew Gelman says he is not a fan of Bayesian hypothesis testing (see here: http://andrewgelman.com/2009/02/26/why_i_dont_like/), and if I'm not misremembering, I think he also says Frequentist hypothesis testing has also shortcomings.

My question is: can you do stats without hypothesis testing even for (sorry for the repetition) hypothesis tests and to make decisions? Is the solution to rely only on estimation and make decisions based on estimated probabilities? If so, can you point out where to learn about this?

  • 2
    $\begingroup$ You can try doing bootstrap simulations, but it will not be a complete math statistics I think. $\endgroup$ – Alexey Burnakov Jun 2 '18 at 18:17
  • 4
    $\begingroup$ I'm perhaps less far over on the spectrum than Gelman but I'd have to say that hypothesis testing is pretty rarely a good way to answer most of the questions it's used to address (we get so many questions phrased as estimation problems, which end with "which test should I use?"... it makes me so sad that people can't even see that they didn't ask a remotely 'test-like' question; many papers look similar). Often the real questions are squarish pegs relentlessly hammered into the roundish hole of a hypothesis test until you no longer notice that they weren't the same shape as when you started. $\endgroup$ – Glen_b Jun 3 '18 at 7:16
  • $\begingroup$ @Glen_b I have tried to present results (which were at times compelling, and other times not) with just estimates and graphics. It's pretty common that it's countered with, "But how do we know it's [significant/meaningful/verifiable]?" To which it's never adequate to say, "Look at this boxplot. It is." On the other hand, if you present $p<0.05$ nobody ever asks the converse "How do we know the effect is relevant?" I think it's a paradox that's being driven largely by non-statisticians. $\endgroup$ – AdamO Jun 3 '18 at 12:54
  • $\begingroup$ I understand the difficulty; the attitude is definitely part of the difficulty of getting people to at least avoid testing when they don't have a question that a test would answer. One could proceed to give standard errors (in large samples) and/or intervals to demonstrate some estimated effect is not simply a result of random variation. It does make me wonder whether the people that say that stuff really think their point nulls are actually going to be true (if they believe in testing they should probably be doing equivalence tests at least). $\endgroup$ – Glen_b Jun 3 '18 at 13:10
  • $\begingroup$ I'll add two points: hypothesis testing is falsely believed to be a major part of statistics because it takes up such a disproportionate amount of statistics teaching. It's ridiculously counterintuitive, and the philosophical backflips that justify it lead the survivors to believe it's critical to any data analysis. Secondly, any decision theoretic framework will lead to false positives and false negatives: all we can do is maximize power and quantify type I error rates. $\endgroup$ – AdamO Jun 4 '18 at 18:36

Let me take the liberty to rephrase the question as "What are the arguments that Andrew Gelman puts forward against hypothesis testing?"

In the paper that is linked in the post, the authors take issue with using a mechanical procedure for model selection, or, as they phrase it:

[Raftery] promises the impossible: The selection of a model that is adequate for specific purposes without consideration of those purposes.

Frequentist or Bayesian hypothesis testing are two examples of such mechanical procedures. The specific method that they criticize is model selection by BIC, which is related to Bayesian hypothesis testing. They list two main cases when such procedures can fail badly:

  1. "Too many data": Say you have a regression model $y_i = \beta'x_i + \epsilon_i$ with, say, 100 standard normally distributed regressors. Say that the first entry of $\beta$ is $1$ and all other entries are equal to $10^{-10}$. Given enough data, a hypothesis test would yield that all estimates of $\beta$ are "significant". Does this mean that we should include $x_2,x_3,\ldots x_{100}$ in the model? If we were interested in discovering some relationships between feature and outcome, would we not be better off considering a model with only $x_1$?
  2. "Not enough data": On the other extreme, if sample sizes are very small, we will be unlikely to find any "significant" relationships. Does this mean that the best model to use is the one that includes no regressors?

There are no general answers to these questions as they depend on the modeler's objective in a given situation. Often, we can try to select models based on criteria that are more closely related to our objective function, e.g. cross validation sample when our objective is prediction. In many situations, however, data-based procedures need to be complemented by expert judgment (or by using the Bayesian approach with carefully chosen priors that Gelman seems to prefer).

  • $\begingroup$ Actually, with regard to point 1, much of machine learning has been interested in this problem: can you create a strong predictor from many weak predictors? I think there is some legitimate promise here. For instance, GWAS studies have honed down the possible genetic contributors to diabetes to somewhere between 20 and 100 SNPs. None of these are as remarkably prognostic as has been previously discovered in other heritable disease (say the BRCA genes and their almost deterministic relationship with breast cancer). This discovery discourages usual approaches to gene therapy for prevention. $\endgroup$ – AdamO Jun 4 '18 at 13:27
  • $\begingroup$ This is a good point. The availability of a general and automatic procedure that creates strong predictions would reduce the analyst's role much further and maybe even eliminate it in many contexts. $\endgroup$ – Matthias Schmidtblaicher Jun 4 '18 at 16:27

The Neyman-Pearson decision-theoretic approach to hypothesis testing (reject/accept) is closely aligned with Popper's Falsification. This method is not invalid, it just has not accommodated the growing human greed for consumption of knowledge, products, and professional gain.

The validity of Popper's approach to science is strongly based on 1. Prespecifying hypotheses 2. Only conducting research with adequate power and 3. Consuming the results of positive/negative studies with equal earnest. We have (in academia, business, government, media, etc) over the past century done none of that.

Fisher proposed a way of doing "stats without hypothesis tests". He never suggested that his p-value be compared to a 0.05 cut-off. He said to report the p-value, and report the power of the study.

Another alternative suggested by many is to merely report the confidence intervals (CIs). The thought is that forcing one to evaluate a trial's results based on a physical quantity, rather than a unitless quantity (like a p-value), would encourage them to consider more subtle aspects like effect size, interpretability, and generalizability. However, even this has fallen flat: the growing tendency is to inspect whether the CI crosses 0 (or 1 for ratio scales) and declare the result statistically significant if not. Tim Lash calls this backdoor hypothesis testing.

There are meandering and endless arguments about a new-era of hypothesis testing. None have not addressed the greed I spoke of earlier. I am of the impression we don't need to change how we do statistics, we need to change how we do science.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.