Let's say the results for an experiment give a p-Value of 0.0354234.

Why is it necessary to fix a threshhold before doing the experiment and than report significance? Why do I not simply report the p-Value given above?

An additional advantage would be that I could repeat the experiment and give summary statistics of the p-Value like mean, median, min, max, and standard deviation. Why is this not common practice?

  • It is a good idea to report the p-value, but if you want to decide to reject $H_0$ or not, then you will need a threshold, so without threshold you don't have a decision ? see e.g. stats.stackexchange.com/questions/166323/… – user83346 Aug 10 '17 at 7:41
  • @fcop Why do I have to make a decision? Why can't I simply state: "The p-Value is 0.0354234. This makes it very unlikely that the assumption of a true Null hypothesis holds true. Therefore, we reject it." I am aware that "very unlikely" is an arbitrary term but setting an arbitrary threshold for the p-Value is not less arbitrary. – Julian Aug 22 '17 at 8:52
  • 1
    The p-value is related to the sample you take, with respect to your comment it means that 3.54234% of an (infinitely) large number of samples will have a value at least as extreme as yours. But the goal of a hypothesis test is to accept or to reject $H_0$ based on one sample. – user83346 Aug 22 '17 at 19:31

It is considered bad practice to pick the significance level (or $\alpha$) post-simulation.

Two reasons for picking the confidence level beforehand:

  1. The significance level is one criterion often used in deciding on an appropriate sample size. See e. g. here.
  2. The analyst is not tempted to choose a cut-off on the basis of what he or she hopes is true. Source

Jim Frost phrases the second point nicely: "It protects you from choosing a significance level because it conveniently gives you significant results!". For a graphical example and further elaboration see his post.

So reporting the p-values for relevant parameters makes sense and should be done (after picking the significance level for your study/case). But just always adding them "because you can" doesn't make sense, consider what information is gained from the reported parameters.

Here some more background on significance levels and reasons to consider them carefully for each case:

"No scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas.” - Fisher (1956 in Statistical Methods and Scientific Inference, p. 42)


The University of Texas in Austin has some nice webpages on the topic, from which I will quote:

It is important to consider the implications and possible consequences of Type I and Type II errors before beginning with data analysis. So it is also a consideration between practical statistical significance. Consider the following example for the difference between the two.

A large clinical trial is carried out to compare a new medical treatment with a standard one. The statistical analysis shows a statistically significant difference in lifespan when using the new treatment compared to the old one. But the increase in lifespan is at most three days, with average increase less than 24 hours, and with poor quality of life during the period of extended life. Most people would not consider the improvement practically significant.

Now back to Type I and Type II erors and why their consideration is so important. (Here a small recap on Type I and Type II errors from datasciencedojo.com:)

enter image description here

Again, this can probably best be explained with examples (also from Texas University).

  • If the consequences of a type I error are serious or expensive, then a very small significance level is appropriate.

Example 1: Two drugs are being compared for effectiveness in treating the same condition. Drug 1 is very affordable, but Drug 2 is extremely expensive. The null hypothesis is "both drugs are equally effective," and the alternate is "Drug 2 is more effective than Drug 1." In this situation, a Type I error would be deciding that Drug 2 is more effective, when in fact it is no better than Drug 1, but would cost the patient much more money. That would be undesirable from the patient's perspective, so a small significance level is warranted.


  • If the consequences of a Type I error are not very serious (and especially if a Type II error has serious consequences), then a larger significance level is appropriate.

Example 2: Two drugs are known to be equally effective for a certain condition. They are also each equally affordable. However, there is some suspicion that Drug 2 causes a serious side-effect in some patients, whereas Drug 1 has been used for decades with no reports of the side effect. The null hypothesis is "the incidence of the side effect in both drugs is the same", and the alternate is "the incidence of the side effect in Drug 2 is greater than that in Drug 1." Falsely rejecting the null hypothesis when it is in fact true (Type I error) would have no great consequences for the consumer, but a Type II error (i.e., failing to reject the null hypothesis when in fact the alternate is true, which would result in deciding that Drug 2 is no more harmful than Drug 1 when it is in fact more harmful) could have serious consequences from a public health standpoint. So setting a large significance level is appropriate.

One rationale for why it makes sense to fix a threshold up-front is that when you determine the sample size based on null hypothesis testing, you would need to use some significance level. Whether the reporting of the results should be so closely tied to and focussed on significant vs. non-significant is of course much more controversial. Many statisticians would disagree with that being a good practice. We are all (hopefully) aware nowadays how important it is that studies with p>0.05 also get published, because otherwise the literature ends up being biased (file drawer problem) and people end up being tempted into research misconduct (i.e. p-hacking their way to a publishable finding = p<=0.05).

In fact, most sensible journals require you to report the full p-value (plus other information), even if p>0.05. There are some journals left in this world that do stupid things like reporting "NS" when "p>0.05", one asterisk for "0.01 < p <= 0.05", two asterisks for "0.001 < p <= 0.01" and three asterisks for "p<=0.001" and similar practices that simply throw away information. I do not think you would find many people that would defend that kind of practice nowadays. Perhaps there was some excuse for it back when you hand-calculated a test statistic and compared it to a table in a book that just gave significance thresholds?! But even then I would have hoped to see the value of the test statistic.

However, I do not think summarizing means/medians/max/min etc. of p-values makes a lot of sense. For summarizing the evidence meta-analysis methods based on e.g. the estimate of an effect size and its standard error (that's a simple vanilla case, often you need to do something else) are much more useful.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.