67
$\begingroup$

In February 2016, the American Statistical Association released a formal statement on statistical significance and p-values. Our thread about it discusses these issues extensively. However, no authority has come forth to offer a universally recognized effective alternative--until now. The American Statistical Society (ASS) has published its response, p-values: What’s next?

"The p-value isn't good for much."

We think the ASA did not go far enough. It is time to admit that the era of p-values is over. Statisticians have successfully used them to baffle undergraduates, trick scientists, and fool editors everywhere, but the world is starting to see through this ruse. We need to abandon this early 20th century attempt by statisticians to control decision making. We need to return to what actually works.

The official ASS proposal is this:

In place of p-values, the ASS advocates the STOP (SeaT-Of-Pants procedure). This time-honored and -tested method was used by the ancient Greeks, renaissance men, and all scientists until Ronald Fisher came along and ruined things. The STOP is simple, direct, data-driven, and authoritative. To carry it out, an authority figure (an older male, by preference) reviews the data and decides whether they agree with his opinion. When he decides they do, the result is “significant.” Otherwise it is not and everybody is required to forget about the whole thing.

Principles

The response addresses each of the ASA's six principles.

  1. The STOP can indicate how incompatible the data are with a specified statistical model.

    We like this phrase because it’s such a fancy way of saying the STOP will answer any question yes or no. Unlike p-values or other statistical procedures, it leaves no doubt. It’s the perfect response to those who say “we don’t need no stinkin’ null hypothesis! What the *?!@ is that, anyway? Nobody ever could figure out what it was supposed to be.”

  2. The STOP doesn’t measure the probability that a hypothesis is true: it actually decides whether it’s true or not.

    Everybody is confused by probabilities. By taking probability out of the picture, the STOP eliminates the need for years of undergraduate and graduate study. Now anybody (who is sufficiently old and male) can perform statistical analysis without the pain and torture of listening to even a single statistical lecture or running arcane software that spews unintelligible output.

  3. Scientific conclusions and business or policy decisions can be based on common sense and real authority figures.

    Important decisions always have been made by authorities, anyway, so let’s just admit it and cut out the middlemen. Using the STOP will free statisticians to do what they are best suited for: using numbers to obfuscate the truth and sanctifying the preferences of those in power.

  4. Proper inference requires full reporting and transparency.

    The STOP is the most transparent and self-evident statistical procedure ever invented: you look at the data and you decide. It eliminates all those confusing z-tests, t-tests, chi-squared tests, and alphabet soup procedures (ANOVA! GLM! MLE!) used by people to hide the fact they have no clue what the data mean.

  5. The STOP measures the importance of the result.

    This is self-evident: if a person in authority employs the STOP, then the result must be important.

  6. By itself, the STOP provides a good measure of evidence regarding a model or hypothesis.

    We wouldn’t want to challenge an authority, would we? Researchers and decision makers will recognize that the STOP provides all the information they need to know. For these reasons, data analysis can end with the STOP; there is no need for alternative approaches, like p-values, machine learning, or astrology.

Other approaches

Some statisticians prefer so-called “Bayesian” methods, in which an obscure theorem posthumously published by an 18th century cleric is applied mindlessly to solve every problem. Its most noted advocates freely admit these methods are “subjective.” If we’re going to use subjective methods, then obviously the more authoritative and knowledgeable the decision maker is, the better the result will be. The STOP thereby emerges as the logical limit of all Bayes methods. Why go to the effort of working those awful calculations, and tying up so much computer time, when you can just show the data to the guy in charge and ask him what his opinion is? End of story.

Another community has recently arisen to challenge the priesthood of statisticians. They call themselves “machine learners” and “data scientists,” but they’re really just hackers looking for higher status. It’s the official position of the ASS that these guys should go form their own professional organization if they want people to take them seriously.


The question

Is this the answer to the problems the ASA identified with p-values and null hypothesis testing? Can it really unite the Bayesian and Frequentist paradigms (as implicitly claimed in the response)?

$\endgroup$
  • 11
    $\begingroup$ "Donald Trump for supreme STOP ASS judge: make statistics great again!" $\endgroup$ – Alex R. Mar 31 '16 at 22:35
  • 14
    $\begingroup$ Clearly STOP is a suboptimal procedure. I am surprised that this has escaped such an esteemed organization of scholars as the ASS. To wit, why waste time looking at the data at all? Simply give your yes/no response. This methodology is already currently in use to great effect. Case studies abound, particularly in the United States during years divisible by 4. $\endgroup$ – cardinal Mar 31 '16 at 23:05
  • 4
    $\begingroup$ I think businesses could also benefit tremendously from adopting these methods, since they'd no longer have to shoulder the heavy cost of hiring people to analyze their data. $\endgroup$ – dsaxton Mar 31 '16 at 23:15
  • 4
    $\begingroup$ @henry As if the [april-1] tag didn't tell us that? $\endgroup$ – Glen_b Apr 1 '16 at 1:09
  • 9
    $\begingroup$ @Henry Seriously? Can you show us any fake organization that gets over a quarter million hits when you Google its name? $\endgroup$ – whuber Apr 1 '16 at 1:50
18
$\begingroup$

I've been advocating for my own new approach to statistical decision making called RADD: Roll A Damn Die. It also addresses all the key points.

1) RADD can indicate how compatible the data are with a specified statistical model.

If you roll a higher number, clearly the evidence is more in favor of your model! An extra benefit is that, if we desire even more confidence, we can roll a die with more sides. You can even find 100 sided dice if you search enough!

2) RADD can decide whether a hypothesis is true or not.

You only have to roll a 2 sided die, i.e., flip a coin.

3) RADD can be used to make business or policy decisions

Get a bunch of policy makers in a room, and have them all roll dice! Highest wins!

4) RADD is transparant.

The result can be recorded, and the die itself can be kept for further research*

5) RADD measures the importance of the result.

Obviously, rolling higher signifies a very important event has occurred.

6) RADD provides a good measure of evidence.

Didn't we say higher rolls are better?

So, no, STOP is not the answer. The answer is RADD.

$\endgroup$
  • 7
    $\begingroup$ Not to forget, it can ensure type I error control (at any desired level given a sufficiently sided dice) e.g. by only rejecting a null hypothesis when one of the 5 sides with the highest number of a 100 sided dice comes up in order to achieve a 5% type I error rate. $\endgroup$ – Björn Apr 1 '16 at 6:27
  • 1
    $\begingroup$ You forgot to mention cool Platonic polyhedra and some cool non-Platonic polyhedra. $\endgroup$ – Alexis Mar 27 at 21:23
17
$\begingroup$

I must say from my experience that in business reality STOP is the default decision making criteria, preferred to $p$-values and other frequentist, or Bayesian methods. From business perspective STOP provides simple and definitive answers what makes it more reliable than uncertain "probabilistic" methods. Moreover, in vast majority of cases it is simpler to implement and easier to adapt to changing reality than other methods. The Yes/No decisions are more convincing for middle and senior management. The "STOP reports" in most cases are shorter and easier to read than the data-based ones. Moreover, adopting this method enables your employer to cut costs on data scientists and SAS licenses. I would say that the only problem with STOP is that it is harder to make PowerPoint presentation presenting STOP results, but this is a dynamically developing field, so in the future better visualization methods may be proposed.

$\endgroup$
  • 6
    $\begingroup$ Once the PowerPoint slides with the conclusion have been briefed, it's too late to change it, so there are two choices, make the analysis fit the conclusion, or don't bother doing analysis at all. $\endgroup$ – Mark L. Stone Apr 1 '16 at 12:02
  • 12
    $\begingroup$ @MarkL.Stone Sure! I personally like the idea of making plots for the presentation before seeing the data, the idea is rooted in Bayesian thinking and I call them a priori plots :) I think that this approach first appeared in print in here: dilbert.com/strip/2008-05-08 $\endgroup$ – Tim Apr 1 '16 at 12:06
15
$\begingroup$

This fine adjunct to the p-value debate, interesting but also somewhat stale in my opinion, reminds me of a unique paper published some years ago in the Christmas issue of the British Medical Journal (BMJ), which every Christmas publishes real yet funny research articles. In particular, this work by Isaacs and Fitzgerald highlighted seven key alternatives to evidence based medicine (ie the practice of medicine based on actual clinical and statistical evidence):

  • Eminence based medicine
  • Vehemence based medicine
  • Eloquence based medicine
  • Providence based medicine
  • Diffidence based medicine
  • Nervousness based medicine
  • Confidence based medicine

Most interestingly, you must look at the columns highlighting the measuring devices and units of measurements for the items above (eg audiometer and decibels for vehemence based medicine!).

$\endgroup$
  • 4
    $\begingroup$ +1. Thank you for a splendid contribution, perfectly in the spirit of the question. (1) Just to clarify: is it the p-value debate you find "stale" or just this question? (2) Would you know where to find reference (6), "J Exponential Salaries"? I'm sure it would have many avid readers if it were better known. $\endgroup$ – whuber Mar 31 '16 at 22:47
  • 5
    $\begingroup$ (1) Your contribution is 'certified fresh' (to quote rottentomatoes.com). Conversely, I find somewhat stale this emphasis on the limitations of p values. In an era of machine learning, big data, and poor scientific literacy among the public, the ASA stance might seem a bit masochist. (2) I think you will find that article in the same journal where they published the randomized trial recommended in this other BMJ Christmas piece: bmj.com/content/327/7429/1459. $\endgroup$ – Joe_74 Mar 31 '16 at 22:57
  • $\begingroup$ I always forget, is it Confidence Based Medicine that uses Dunning-Kruger-based inference? $\endgroup$ – Alexis Mar 20 '18 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.