28
$\begingroup$

I'm into epidemiology. I'm not a statistician but I try to perform the analyses myself, although I often encounter difficulties. I did my first analysis some 2 years ago. P values were included everywhere in my analyses (I simply did what other researchers were doing) from descriptive tables to regression analyses. Little by little, statisticians working in my apartment persuaded me to skip all (!) the p values, except from where I truly have a hypothesis.

The problem is that p values are abundant in medical research publications. It is conventional to include p values on far too many lines; descriptive data of means, medians or whatever usually go along with p values (students t-test, Chi-square etc).

I've recently submitted a paper to a journal, and I refused (politely) to add p values to my "baseline" descriptive table. The paper was ultimately rejected.

To exemplify, see the figure below; it is the descriptive table from the latest published article in a respected journal of internal medicine.: enter image description here

Statisticians are mostly (if not always) involved in the reviewing of these manuscripts. So a laymen like myself expects to not find any p values where there are no hypothesis. But they are abundant, but the reason for this remain elusive to me. I find it hard to believe that it is ignorance.

I realize that this is a borderline statistical question. But I'm looking for the rationale behind this phenomenon.

$\endgroup$
  • 12
    $\begingroup$ A p-value without a hypothesis is inherently flawed. What does a p-value even mean when you don't have a hypothesis? $\endgroup$ – jameselmore Mar 19 '15 at 22:18
  • 3
    $\begingroup$ Can you perhaps give some examples of people using p-values without any hypothesis? This is not clear. $\endgroup$ – amoeba says Reinstate Monica Mar 19 '15 at 22:41
  • 4
    $\begingroup$ @amoeba ""The problem is that p values are everywhere in every medical journal. It is conventional to include p values on every line where there are means, medians or proportions described."" They tend to be simple Fisher exact tests or chi-square tests for differences, asking if any row of a summary table has a significant difference. The implied hypothesis is that each row matters. $\endgroup$ – Karl Mar 19 '15 at 22:56
  • 2
    $\begingroup$ I suspect a major force is that p-values give a misleading impression of finality to a given claim. The publishers of these journals should love this since it means they own information that will be valuable for the foreseeable future. The concurrent culture of not funding or proposing replication studies also helps to minimize the presence of controversial conflicting results. I wonder what will happen if people eventually realize the information they own consists mostly of "pointless activity" (@glen_b's term). Even if there is useful stuff mixed in...heuristics tell you to avoid. $\endgroup$ – Livid Mar 20 '15 at 4:22
  • 1
    $\begingroup$ [at]jameselmore: I'm asking the same question; it makes no sense but it is applied every day. [at]amoeba: I randomly choose one of the journals that I read, hit the latest published article and found this: onlinelibrary.wiley.com/doi/10.1111/joim.12230/full [at]Karl: exactly, thank you. @Momo: I have done an effort now to improve the formulation of the question. I think this is an important question and I appreciate your suggestion. [at]Livid: thank you for this comment. Indeed many researchers might have misunderstood the whole point of p values. $\endgroup$ – Adam Robinsson Mar 20 '15 at 17:08
29
$\begingroup$

Clearly I don't need to tell you what a p-value is, or why over-reliance on them is a problem; you apparently understand those things quite well enough already.

With publishing, you have two competing pressures.

The first - and one you should push for at every reasonable opportunity - is to do what makes sense.

The second, ultimately, is the need to actually publish. There's little gain if nobody sees your fine efforts at reforming terrible practice.

So instead of avoiding it altogether:

  • do it as little of such pointless activity as you can get away with that still gets it published

  • maybe include a mention of this recent Nature methods article[1] if you think it will help, or perhaps better one or more of the other references. It at least should help establish that there's some opposition to the primacy of p-values.

  • consider other journals, if another would be suitable

Is this the same in other disciplines?

The problem of over-use of p-values occurs in a number of disciplines (this can even be a problem when there is some hypothesis), but is much less common in some than others. Some disciplines do have issues with p-value-itis, and the problems that causes can eventually lead to somewhat overblown reactions[2] (and to a smaller extent, [1], and at least in some places, a few of the others as well).

I think there are a variety of reasons for it, but the over-reliance of p-values seems to acquire a momentum of its own - there's something about saying "significant" and rejecting a null that people seem to find very attractive; various disciplines (e.g. see [3][4][5][6][7][8][9][10][11]) have (with varying degrees of success) been fighting against the problem of over reliance on p-values (especially $\alpha$=0.05) for many years, and have made many different kinds of suggestions - not all of which I agree with, but I include a variety of views to give some sense of the different things people have had to say.

Some of them advocate focusing on confidence intervals, some advocate looking at effect sizes, some advocate Bayesian methods, some smaller p-values, some just on avoiding using p-values in particular ways, and so on. There are many different views on what to do instead, but between them there's a lot of material on problems with relying on p-values, at least the way it's pretty commonly done.

See those references for many further references in turn. This is just a sampling - many dozens more references can be found. A few authors give reasons why they think p-values are prevalent.

Some of these references may be useful if you do want to argue the point with an editor.

[1] Halsey L.G., Curran-Everett D., Vowler S.L. & Drummond G.B. (2015),
"The fickle P value generates irreproducible results,"
Nature Methods 12, 179–185 doi:10.1038/nmeth.3288
http://www.nature.com/nmeth/journal/v12/n3/abs/nmeth.3288.html

[2] David Trafimow, D. and Marks, M. (2015),
Editorial,
Basic and Applied Social Psychology, 37:1–2
http://www.tandfonline.com/loi/hbas20
DOI: 10.1080/01973533.2015.1012991

[3] Cohen, J. (1990),
Things I have learned (so far),
American Psychologist, 45(12), 1304–1312.

[4] Cohen, J. (1994),
The earth is round (p < .05),
American Psychologist, 49(12), 997–1003.

[5] Valen E. Johnson (2013),
Revised standards for statistical evidence PNAS, vol. 110, no. 48, 19313–19317 http://www.pnas.org/content/110/48/19313.full.pdf

[6] Kruschke J.K. (2010),
What to believe: Bayesian methods for data analysis,
Trends in cognitive sciences 14(7), 293-300

[7] Ioannidis, J. (2005)
Why Most Published Research Findings Are False,
PLoS Med. Aug; 2(8): e124.
doi: 10.1371/journal.pmed.0020124

[8] Gelman, A. (2013), P Values and Statistical Practice,
EpidemiologyVol.24, No. 1, January, 69-72

[9] Gelman, A. (2013),
"The problem with p-values is how they're used",
(Discussion of “In defense of P-values,” by Paul Murtaugh, for Ecology) unpublished
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.300.9053
http://www.stat.columbia.edu/~gelman/research/unpublished/murtaugh2.pdf

[10] Nuzzo R. (2014),
Statistical errors: P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume,
News and Comment,
Nature, Vol. 506 (13), 150-152

[11] Wagenmakers E, (2007)
A practical solution to the pervasive problems of p values,
Psychonomic Bulletin & Review 14(5), 779-804

$\endgroup$
  • 7
    $\begingroup$ +1. I read this Nature Methods paper [1] another week and I am not sure I like it very much. They essentially argue that p-values can be very variable in low power tests (see also "dance of p-values" on youtube) -- something that is of course true and that does need to be emphasized. They conclude that p-values are "bad" (the title sounds pretty harsh) and that people should use confidence intervals which are "good". But of course confidence intervals are also very variable in low power! The situation on their Figure 6 (left) does not look much better to me than on Figure 2. $\endgroup$ – amoeba says Reinstate Monica Mar 20 '15 at 10:14
  • 2
    $\begingroup$ @amoeba I won't say I disagree with you - there's quite a lot there I disagree with; nevertheless there are some points there that may be useful to the OP. Actually, you've reminded me of a change I intended to make but forgot about. $\endgroup$ – Glen_b Mar 20 '15 at 10:17
  • 3
    $\begingroup$ Yes, I agree with potential usefulness nevertheless -- especially because Nature Methods is respectable enough that people can perhaps be convinced by its "authority". I justed wanted to warn OP against taking everything there for granted (their math is okay, I am talking about conclusions/interpretation here). $\endgroup$ – amoeba says Reinstate Monica Mar 20 '15 at 10:21
  • 1
    $\begingroup$ Also interesting in this context is Wilkinson and the Task Force on Statistical Inference, Statistical Methods in Psychology Journals, American Psychologist, Vol. 54, No. 8, 594-604, 1999. $\endgroup$ – A. Donda Mar 22 '15 at 15:35
  • $\begingroup$ Glen_b, I posted a question about one of the stranger claims in the "Fickle P" paper: stats.stackexchange.com/questions/250269 - would greatly appreciate your insight. $\endgroup$ – amoeba says Reinstate Monica Dec 8 '16 at 12:43
10
$\begingroup$

The p-value, or more generally, null-hypothesis significance testing (NHST), is slowly holding less and less value. So much so that is has started to get banned in journals.

Most people don't understand what the p-value really tells us and why it tells us this, even though it is used everywhere.

The problem is that the p-value tells us $P(\text{Data}\,\vert\, H_0)$ and not $P(H_0\,\vert\,\text{Data})$, which is the more informative one. The latter involves the use the Bayesian inference, and provides a stronger basis for conclusions of model checking.

The probability of the $H_0$ model being true/significant, given the data we have observed, has stronger implications than the probability of our data fitting the $H_0$ model.

$\endgroup$
  • 1
    $\begingroup$ I would add that P(H0|data) is only meaningful if H0 is meaningful. The studies need to be designed and reported in a way to rule out other uninteresting explanations for the results (bias, dropouts, baseline differences) beyond chance. Also, even a perfect blinded RCT with substantial effect size only tells you that something interesting was measured. Figuring out if you measured the thing you are actually concerned with is another glossed over issue often found along with the p-value obsession. $\endgroup$ – Livid Mar 20 '15 at 4:55
8
$\begingroup$

Is this the same in other disciplines? What is the reason for the obsession with p values?

Greenwald et al. (1996) attempt to deal with this question regarding psychology. As to also applying NHST to baseline differences, presumably the editors will (rightly or wrongly) decide that "non-significant" baseline differences cannot explain the results, while "significant" ones may explain the results. This is similar to "Reason 1" offered by Greenwald et al. :

Why Does NHT Remain Popular?

"Why does NHT not succumb to criticism? For lack of a better answer, it is tempting to credit the persistence of NHT to behavioral scientists' lack of character. Behavioral scientists' unwillingness to renounce the guilty pleasure of obtaining possibly spurious null hypothesis rejections may be like a drinker's unwillingness to renounce the habit of a pre-dinner cocktail..."

Reason I: HT Provides a Dichotomous Outcome

"Because of widespread adoption of the convention that p < .05 translates to "statistically significant," NHT can be used to yield a dichotomous answer (reject or don't reject) to a question about a null hypothesis. This may often be regarded as a useful answer for theoretical questions that are stated in terms of a direction of prediction rather than in terms of the expected value of a parameter..."

Reason 2: p Value as a Meaningful Common- Language Translation for Test Statistics

"Unlike anything that can be perceived so directly from t, F, or r values (with their associated df ), a p value's measure of surprise is simply captured by the number of consecutive zeros to the right of its decimal point..."

Reason 3: p Value Provides a Measure of Confidence" in Replicability of Null Hypothesis Rejections

"[U]nlike an effect size (or a confidence interval), a p value resulting from NHT is monotonically related to an estimate of a non-null finding's replicability. In this statement, replicability (which is defined more formally just below) is intended only in its NHT sense of repeating the reject-nonreject conclusion and not in its estimation sense of proximity between point or interval estimates."

Effect sizes and p values: What should be reported and what should be replicated? ANTHONY G. GREENWALD, RICHARD GONZALEZ, RICHARD J. HARRIS, AND DONALD GUTHRIE. Psychophysiology, 33 (1996). 175-183. Cambridge University Press. Printed in the USA. Copyright O 1996 Society for Psychophysiological Research

$\endgroup$
  • $\begingroup$ thank you for these important comments, which I will definitely use to argue with reviewers next time. $\endgroup$ – Adam Robinsson Mar 21 '15 at 9:17
6
$\begingroup$

P-values give information about differences between two groups of results ("treatment" vs "control", "A" vs "B", etc.) that sample from two populations. The nature of the difference is formalized in the statement of hypotheses -- e.g. "mean of A is greater than mean of B". Low p-values suggest that the differences are not due to random variation, while high p-values suggest that differences in the two samples cannot be distinguished from differences that might arise simply from random variation. What is "low" or "high" for a p-value has historically been a matter of convention and taste rather than established by rigorous logic or analysis of evidence.

A prerequisite for using p-values is that the two groups of results are really comparable, namely that the only source of difference between them is related to variable you are evaluating. As a exaggerated example, imagine that you have statistics on two diseases in two time periods -- A: mortality from cholera among men in British prisons 1920-1930, and B: infection by malaria in Nigeria 1960-1970. Computing a p-value from these two sets of data would be rather absurd. Now, if A: mortality from cholera among men in British prisons who are not treated vs. B: mortality from cholera among men in British prisons treated with re-hydration, then you have the basis for a solid statistical hypothesis.

Most often this is accomplished through careful experiment design, or careful survey design, or careful collection of historical data, etc. Also, the differences between the two results must be formalized into hypotheses statements involving sample statistics -- often sample means, but could also be sample variances, or other sample statistics. It's also possible to create hypotheses statements comparing the two sample distributions as a whole, using stochastic dominance. These are rare.

The controversy over p-values centers on "what is really significant" for research? This is where effect sizes come in. Basically, effect size is the magnitude of the difference between the two groups. It's possible to have high statistical significance (low p-value -> not due to random variation) but also low effect size (very little difference in magnitude). When effect sizes are very large, then allowing somewhat high p-values may be OK.

Most disciplines are now moving very strongly toward reporting effect sizes, and reducing or minimizing the role of p-values. They also encourage more descriptive statistics about the sample distributions. Some approaches, including Bayesian Statistics, do away with p-values all together.


My answer is condensed and simplified. There are many articles on this topic you can consult for more details, justifications, and specifics, including these:

$\endgroup$
  • $\begingroup$ @MerMeritology thank you for providing these important references. I'll read them ASAP! $\endgroup$ – Adam Robinsson Mar 21 '15 at 9:18
6
$\begingroup$

"So a laymen like myself expects to not find any p values where there are no hypothesis."

Implicitly, the OP says that in the specific Table he presents, there are no hypotheses that accompany the reported p-values. Just to clear away this small confusion, there certainly are null hypotheses, but they are rather... indirectly mentioned (for economy of space, I presume).

The "p-value" is a conditional probability, say, for a "right-tail" test,

$$\text{p-val} \equiv P(T\geq t(S) \mid H_0) = 1-F_{T|H_0}(t(S) \mid H_0)$$

where $T$ is the statistic used, $F_{T|H_0}(t \mid H_0)$ is the cummulative distribution function that characterizes the probabilities related to $T$ conditional on $H_0$ being true, and $t(S)$ is the value of $T$ obtained by the use of the sample at hand. Obviously, for the test to be meaningful, it must be the case that the statistic $T$ is such and the null hypothesis $H_0$ is such that the distribution of $T$ conditional on $H_0$ being true, is different (or differently parametrized, when they both belong to the same family) from its distribution conditional on $H_0$ not being true.

So a p-value cannot even be calculated if there is no null hypothesis, and whenever we see a p-value reported, somewhere there a null hypothesis lurks.

In the Table presented in the question we read

"All tests for differences across WHR tertiles..."

The null-hypothesis is "hidden" in this phrase: it is "No difference between WHR tertiles", (whatever a "WΗR tertile" is) expressed in its mathematical form which here appears to be a difference of two magnitudes being set equal to zero.

$\endgroup$
  • $\begingroup$ I agree there could be hypotheses behind these analyses. However, those who elaborate guidelines for research papers (e.g STROBE statement) should adress the abundance of p values. I think a p value should be reserved for the main hypothesis of a paper (which is rarely more than one). But nevertheless, I cannot say I disagree with you =) $\endgroup$ – Adam Robinsson Mar 21 '15 at 21:20
  • 1
    $\begingroup$ @AdamRobinsson Hmmm... I am not so sure. Such a "reserved" approach, would inflate (even more) the importance that a p-value test really has for reaching a conclusion. To me, it is just one more result that has to be combined with many other aspects, results, out-of-sample information, logic, etc. On the other hand, if p-values are scattered all over the place, it is easier to realize that they are not the definite criterion to reach conclusions. $\endgroup$ – Alecos Papadopoulos Mar 22 '15 at 23:08
  • $\begingroup$ Alecos I read something different in the table, which refers to WHR (i.e. waist-hip ratio) tertiles rather than WRT, while tertiles are values that divide a distribution into 3 parts in the same sense that quartiles are values that divide into 4 parts and deciles are to ten parts. $\endgroup$ – Glen_b Mar 24 '15 at 0:28
  • $\begingroup$ @Glen_b Thanks, that was just a typo from my part. Fixed it. $\endgroup$ – Alecos Papadopoulos Mar 24 '15 at 0:42
  • 2
    $\begingroup$ See, for example, here. But probably not here. $\endgroup$ – Glen_b Mar 24 '15 at 0:47
2
$\begingroup$

I got curious and read the paper that OP gave as an example: Abdominal obesity increases the risk of hip fracture. I am not a medical researcher and normally do not read medicine papers.

I was surprised to see that the ONLY place where this paper uses $p$-values is the caption of Table 1 that OP reproduced in the question body.

To me it does not look like an "abundance" of $p$-values at all! I am used to neuroscience papers, where different groups of subjects (humans, mice, flies, neurons, tissue samples, etc.) get differently treated or measured in different conditions, and papers usually revolve around the differences between groups. These differences are always assessed with $p$-values, so a paper can have dozens and dozens of them reported in the main text. At times, this really does look like "an abundance". This approach is often (sometimes rightly and sometimes wrongly) criticized for various reasons, see an answer by @Glen_b (+1) and further links.

However, this paper does not do anything like that and only reports $p$-values basically in the introduction, when different characteristics of the cohort are reported. I don't understand what the $p$-values are doing there, and so yes, I agree that they are out of place. However, I don't understand what this whole table is doing there either! I find this table rather confusing (why tertiles? why tertiles of WHR? where is the actual variable of interest, the hip fracture rate?) and it does not seem to be used for any actual analysis further on. This whole table could be kicked out of the text without much loss, together with the $p$-values.

As I do not see any abundance of $p$-values in this paper, I am somewhat confused by the question.

It sounds as if the question is specifically referring to such descriptive tables. If so, this is some weird (but mostly harmless?) practice in medical journals, surviving due to tradition.


P.S. By the way, the main analysis of this paper (that does not involve any $p$-values) looks weird to me. The goal of the study is "to examine [...] the relationship between waist circumference (WC), hip circumference (HC), waist/hip ratio (WHR) and BMI to incident hip fracture", while controlling for various possible covariates. Sample size is huge ($n=43000$). What I would do, is to put all predictors into one regression model with an elastic net penalty, select the regularization parameters via cross-validation, and then look at what predictors have non-zero coefficients. Or something similar. The authors, instead, do some ad hoc modeling.

$\endgroup$
  • $\begingroup$ @amoeba I selected an article at rando; it was the latest published article in epidemiology in that journal. I'm sure if I had searched some more I could have provided an article with many more pointless p values. As you have noticed, there is a p-valueitis but from your, and the other answers above and below, it appears that the research community is addressing this. $\endgroup$ – Adam Robinsson Mar 21 '15 at 9:20
  • $\begingroup$ @Adam, I like your question (+1) and Glen_b's answer (+1), but if this "randomly selected" paper is representative, then most points that Glen_b made and most papers that he linked to, do not apply or refer to the situation in medical research that you were asking about. If it is not representative, then of course I cannot judge. $\endgroup$ – amoeba says Reinstate Monica Mar 21 '15 at 14:31
  • $\begingroup$ I indeed have had immense help from your answers several times. I did the judgement based on my understanding of this problem. I believe all answers provided are useful and they collectively answer the question. $\endgroup$ – Adam Robinsson Mar 21 '15 at 21:17
1
$\begingroup$

The level of statistical peer-review is not as high as one would think from my experience. For all applied papers I have worked on, all of the statistical comments came from experts in the applied field and not from statisticians. For "top" journals, although there is greater scrutiny, it is not uncommon to see results that have serious faults. I think this is partly because the field of statistics can be difficult (as can be seen by disagreements between many of its great minds).

Second, readers in a field expect to see things in a certain way. In one recent experience, I plotted probabilities from a model, but this was shot down because my collaborator guessed correctly this his readers would be more comfortable with a barplot of raw data. In sum, many readers expect to see p-values alongside a table of baseline characteristics.

Unrelated to your direct question, but perhaps relevant: p-values are used in almost every text using frequentist or likelihood methods. The authors often have made tremendous contributions and have thought deeply about statistics. Although abused by experimentalists, surely they have a place in statistics.

$\endgroup$
  • $\begingroup$ thank you for this comment. I could take your statement even further; I think that an unbelievably large proportion of published findings contains statistical flaws for various reasons. My supervisor often says "the review process is based on a gentlemen's word" Quite funny I think. $\endgroup$ – Adam Robinsson Mar 21 '15 at 9:24
1
$\begingroup$

I have to read medical articles often and I feel that the pendulum seems to be swinging from one extreme to another, rather than staying in the central balanced zone.

Following approach seems to work well. If the P value is small, the observed difference is unlikely to be by chance alone. We should, hence, look at the magnitude of the difference and decide whether it is of any practical significance. Very small P values occur with large sample sizes even with very small differences which may be of no practical relevance.

Not including P values in the table of baseline data may be disadvantageous. So if in a study there are two groups with mean ages are 54 and 59 years, I want to know if this difference can be by chance alone. If P is small then I think whether this 5 year difference in 2 groups can affect the results of the study. If P is not small, I do not have to address this question.

Problem occurs if one relies solely on the P value and not check the magnitude of the difference (for example, simple percent change). Some feel that P values should be totally omitted so that only the difference remains and is seen. A balanced solution would be to emphasize on evaluating both these and not to just throw away the P value, which has a limited but 'significant' meaning. The effect size is also likely to correlate closely with P value (just like confidence intervals) and it is also unlikely to completely displace P values from the statistical landscape. As mentioned in following article, there are many virtues of null hypothesis testing because of which it remains popular:

ANTHONY G. GREENWALD, RICHARD GONZALEZ, RICHARD J. HARRIS, AND DONALD GUTHRIE Effect sizes and p values: What should be reported and what should be replicated? Psychophysiology, 33 (1996). 175-183.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.