Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Let's say the results for an experiment give a p-Value of 0.0354234.
Why is it necessary to fix a threshhold before doing the experiment and than report significance? Why do I not simply report the p-Value given above?
An additional advantage would be that I could repeat the experiment and give summary statistics of the p-Value like mean, median, min, max, and standard deviation. Why is this not common practice?
It is considered bad practice to pick the significance level (or $\alpha$) post-simulation.
Two reasons for picking the confidence level beforehand:
Jim Frost phrases the second point nicely: "It protects you from choosing a significance level because it conveniently gives you significant results!". For a graphical example and further elaboration see his post.
So reporting the p-values for relevant parameters makes sense and should be done (after picking the significance level for your study/case). But just always adding them "because you can" doesn't make sense, consider what information is gained from the reported parameters.
Here some more background on significance levels and reasons to consider them carefully for each case:
"No scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case in the light of his evidence and his ideas.” - Fisher (1956 in Statistical Methods and Scientific Inference, p. 42)
The University of Texas in Austin has some nice webpages on the topic, from which I will quote:
It is important to consider the implications and possible consequences of Type I and Type II errors before beginning with data analysis. So it is also a consideration between practical statistical significance. Consider the following example for the difference between the two.
A large clinical trial is carried out to compare a new medical treatment with a standard one. The statistical analysis shows a statistically significant difference in lifespan when using the new treatment compared to the old one. But the increase in lifespan is at most three days, with average increase less than 24 hours, and with poor quality of life during the period of extended life. Most people would not consider the improvement practically significant.
Now back to Type I and Type II erors and why their consideration is so important. (Here a small recap on Type I and Type II errors from datasciencedojo.com:)
Again, this can probably best be explained with examples (also from Texas University).
- If the consequences of a type I error are serious or expensive, then a very small significance level is appropriate.
Example 1: Two drugs are being compared for effectiveness in treating the same condition. Drug 1 is very affordable, but Drug 2 is extremely expensive. The null hypothesis is "both drugs are equally effective," and the alternate is "Drug 2 is more effective than Drug 1." In this situation, a Type I error would be deciding that Drug 2 is more effective, when in fact it is no better than Drug 1, but would cost the patient much more money. That would be undesirable from the patient's perspective, so a small significance level is warranted.
- If the consequences of a Type I error are not very serious (and especially if a Type II error has serious consequences), then a larger significance level is appropriate.
Example 2: Two drugs are known to be equally effective for a certain condition. They are also each equally affordable. However, there is some suspicion that Drug 2 causes a serious side-effect in some patients, whereas Drug 1 has been used for decades with no reports of the side effect. The null hypothesis is "the incidence of the side effect in both drugs is the same", and the alternate is "the incidence of the side effect in Drug 2 is greater than that in Drug 1." Falsely rejecting the null hypothesis when it is in fact true (Type I error) would have no great consequences for the consumer, but a Type II error (i.e., failing to reject the null hypothesis when in fact the alternate is true, which would result in deciding that Drug 2 is no more harmful than Drug 1 when it is in fact more harmful) could have serious consequences from a public health standpoint. So setting a large significance level is appropriate.
One rationale for why it makes sense to fix a threshold up-front is that when you determine the sample size based on null hypothesis testing, you would need to use some significance level. Whether the reporting of the results should be so closely tied to and focussed on significant vs. non-significant is of course much more controversial. Many statisticians would disagree with that being a good practice. We are all (hopefully) aware nowadays how important it is that studies with p>0.05 also get published, because otherwise the literature ends up being biased (file drawer problem) and people end up being tempted into research misconduct (i.e. p-hacking their way to a publishable finding = p<=0.05).
In fact, most sensible journals require you to report the full p-value (plus other information), even if p>0.05. There are some journals left in this world that do stupid things like reporting "NS" when "p>0.05", one asterisk for "0.01 < p <= 0.05", two asterisks for "0.001 < p <= 0.01" and three asterisks for "p<=0.001" and similar practices that simply throw away information. I do not think you would find many people that would defend that kind of practice nowadays. Perhaps there was some excuse for it back when you hand-calculated a test statistic and compared it to a table in a book that just gave significance thresholds?! But even then I would have hoped to see the value of the test statistic.
However, I do not think summarizing means/medians/max/min etc. of p-values makes a lot of sense. For summarizing the evidence meta-analysis methods based on e.g. the estimate of an effect size and its standard error (that's a simple vanilla case, often you need to do something else) are much more useful.
