The abundance of P values in absence of a hypothesis

Question

I'm into epidemiology. I'm not a statistician but I try to perform the analyses myself, although I often encounter difficulties. I did my first analysis some 2 years ago. P values were included everywhere in my analyses (I simply did what other researchers were doing) from descriptive tables to regression analyses. Little by little, statisticians working in my apartment persuaded me to skip all (!) the p values, except from where I truly have a hypothesis.

The problem is that p values are abundant in medical research publications. It is conventional to include p values on far too many lines; descriptive data of means, medians or whatever usually go along with p values (students t-test, Chi-square etc).

I've recently submitted a paper to a journal, and I refused (politely) to add p values to my "baseline" descriptive table. The paper was ultimately rejected.

To exemplify, see the figure below; it is the descriptive table from the latest published article in a respected journal of internal medicine.:

Statisticians are mostly (if not always) involved in the reviewing of these manuscripts. So a laymen like myself expects to not find any p values where there are no hypothesis. But they are abundant, but the reason for this remain elusive to me. I find it hard to believe that it is ignorance.

I realize that this is a borderline statistical question. But I'm looking for the rationale behind this phenomenon.

Answer 1

Clearly I don't need to tell you what a p-value is, or why over-reliance on them is a problem; you apparently understand those things quite well enough already.

With publishing, you have two competing pressures.

The first - and one you should push for at every reasonable opportunity - is to do what makes sense.

The second, ultimately, is the need to actually publish. There's little gain if nobody sees your fine efforts at reforming terrible practice.

So instead of avoiding it altogether:

• do it as little of such pointless activity as you can get away with that still gets it published

• maybe include a mention of this recent Nature methods article[1] if you think it will help, or perhaps better one or more of the other references. It at least should help establish that there's some opposition to the primacy of p-values.

• consider other journals, if another would be suitable

Is this the same in other disciplines?

The problem of over-use of p-values occurs in a number of disciplines (this can even be a problem when there is some hypothesis), but is much less common in some than others. Some disciplines do have issues with p-value-itis, and the problems that causes can eventually lead to somewhat overblown reactions[2] (and to a smaller extent, [1], and at least in some places, a few of the others as well).

I think there are a variety of reasons for it, but the over-reliance of p-values seems to acquire a momentum of its own - there's something about saying "significant" and rejecting a null that people seem to find very attractive; various disciplines (e.g. see [3][4][5][6][7][8][9][10][11]) have (with varying degrees of success) been fighting against the problem of over reliance on p-values (especially $\alpha$=0.05) for many years, and have made many different kinds of suggestions - not all of which I agree with, but I include a variety of views to give some sense of the different things people have had to say.

Some of them advocate focusing on confidence intervals, some advocate looking at effect sizes, some advocate Bayesian methods, some smaller p-values, some just on avoiding using p-values in particular ways, and so on. There are many different views on what to do instead, but between them there's a lot of material on problems with relying on p-values, at least the way it's pretty commonly done.

See those references for many further references in turn. This is just a sampling - many dozens more references can be found. A few authors give reasons why they think p-values are prevalent.

Some of these references may be useful if you do want to argue the point with an editor.

[1] Halsey L.G., Curran-Everett D., Vowler S.L. & Drummond G.B. (2015),
"The fickle P value generates irreproducible results,"
Nature Methods 12, 179–185 doi:10.1038/nmeth.3288
http://www.nature.com/nmeth/journal/v12/n3/abs/nmeth.3288.html

[2] David Trafimow, D. and Marks, M. (2015),
Editorial,
Basic and Applied Social Psychology, 37:1–2
http://www.tandfonline.com/loi/hbas20
DOI: 10.1080/01973533.2015.1012991

[3] Cohen, J. (1990),
Things I have learned (so far),
American Psychologist, 45(12), 1304–1312.

[4] Cohen, J. (1994),
The earth is round (p < .05),
American Psychologist, 49(12), 997–1003.

[5] Valen E. Johnson (2013),
Revised standards for statistical evidence PNAS, vol. 110, no. 48, 19313–19317 http://www.pnas.org/content/110/48/19313.full.pdf

[6] Kruschke J.K. (2010),
What to believe: Bayesian methods for data analysis,
Trends in cognitive sciences 14(7), 293-300

[7] Ioannidis, J. (2005)
Why Most Published Research Findings Are False,
PLoS Med. Aug; 2(8): e124.
doi: 10.1371/journal.pmed.0020124

[8] Gelman, A. (2013), P Values and Statistical Practice,
EpidemiologyVol.24, No. 1, January, 69-72

[9] Gelman, A. (2013),
"The problem with p-values is how they're used",
(Discussion of “In defense of P-values,” by Paul Murtaugh, for Ecology) unpublished
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.300.9053
http://www.stat.columbia.edu/~gelman/research/unpublished/murtaugh2.pdf

[10] Nuzzo R. (2014),
Statistical errors: P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume,
News and Comment,
Nature, Vol. 506 (13), 150-152

[11] Wagenmakers E, (2007)
A practical solution to the pervasive problems of p values,
Psychonomic Bulletin & Review 14(5), 779-804

Answer 2

The p-value, or more generally, null-hypothesis significance testing (NHST), is slowly holding less and less value. So much so that is has started to get banned in journals.

Most people don't understand what the p-value really tells us and why it tells us this, even though it is used everywhere.

The problem is that the p-value tells us $P(\text{Data}\,\vert\, H_0)$ and not $P(H_0\,\vert\,\text{Data})$, which is the more informative one. The latter involves the use the Bayesian inference, and provides a stronger basis for conclusions of model checking.

The probability of the $H_0$ model being true/significant, given the data we have observed, has stronger implications than the probability of our data fitting the $H_0$ model.

Answer 3

Is this the same in other disciplines? What is the reason for the obsession with p values?

Greenwald et al. (1996) attempt to deal with this question regarding psychology. As to also applying NHST to baseline differences, presumably the editors will (rightly or wrongly) decide that "non-significant" baseline differences cannot explain the results, while "significant" ones may explain the results. This is similar to "Reason 1" offered by Greenwald et al. :

Why Does NHT Remain Popular?

"Why does NHT not succumb to criticism? For lack of a better answer, it is tempting to credit the persistence of NHT to behavioral scientists' lack of character. Behavioral scientists' unwillingness to renounce the guilty pleasure of obtaining possibly spurious null hypothesis rejections may be like a drinker's unwillingness to renounce the habit of a pre-dinner cocktail..."

Reason I: HT Provides a Dichotomous Outcome

"Because of widespread adoption of the convention that p < .05 translates to "statistically significant," NHT can be used to yield a dichotomous answer (reject or don't reject) to a question about a null hypothesis. This may often be regarded as a useful answer for theoretical questions that are stated in terms of a direction of prediction rather than in terms of the expected value of a parameter..."

Reason 2: p Value as a Meaningful Common- Language Translation for Test Statistics

"Unlike anything that can be perceived so directly from t, F, or r values (with their associated df ), a p value's measure of surprise is simply captured by the number of consecutive zeros to the right of its decimal point..."

Reason 3: p Value Provides a Measure of Confidence" in Replicability of Null Hypothesis Rejections

"[U]nlike an effect size (or a confidence interval), a p value resulting from NHT is monotonically related to an estimate of a non-null finding's replicability. In this statement, replicability (which is defined more formally just below) is intended only in its NHT sense of repeating the reject-nonreject conclusion and not in its estimation sense of proximity between point or interval estimates."

Effect sizes and p values: What should be reported and what should be replicated? ANTHONY G. GREENWALD, RICHARD GONZALEZ, RICHARD J. HARRIS, AND DONALD GUTHRIE. Psychophysiology, 33 (1996). 175-183. Cambridge University Press. Printed in the USA. Copyright O 1996 Society for Psychophysiological Research

Answer 4

P-values give information about differences between two groups of results ("treatment" vs "control", "A" vs "B", etc.) that sample from two populations. The nature of the difference is formalized in the statement of hypotheses -- e.g. "mean of A is greater than mean of B". Low p-values suggest that the differences are not due to random variation, while high p-values suggest that differences in the two samples cannot be distinguished from differences that might arise simply from random variation. What is "low" or "high" for a p-value has historically been a matter of convention and taste rather than established by rigorous logic or analysis of evidence.

A prerequisite for using p-values is that the two groups of results are really comparable, namely that the only source of difference between them is related to variable you are evaluating. As a exaggerated example, imagine that you have statistics on two diseases in two time periods -- A: mortality from cholera among men in British prisons 1920-1930, and B: infection by malaria in Nigeria 1960-1970. Computing a p-value from these two sets of data would be rather absurd. Now, if A: mortality from cholera among men in British prisons who are not treated vs. B: mortality from cholera among men in British prisons treated with re-hydration, then you have the basis for a solid statistical hypothesis.

Most often this is accomplished through careful experiment design, or careful survey design, or careful collection of historical data, etc. Also, the differences between the two results must be formalized into hypotheses statements involving sample statistics -- often sample means, but could also be sample variances, or other sample statistics. It's also possible to create hypotheses statements comparing the two sample distributions as a whole, using stochastic dominance. These are rare.

The controversy over p-values centers on "what is really significant" for research? This is where effect sizes come in. Basically, effect size is the magnitude of the difference between the two groups. It's possible to have high statistical significance (low p-value -> not due to random variation) but also low effect size (very little difference in magnitude). When effect sizes are very large, then allowing somewhat high p-values may be OK.

Most disciplines are now moving very strongly toward reporting effect sizes, and reducing or minimizing the role of p-values. They also encourage more descriptive statistics about the sample distributions. Some approaches, including Bayesian Statistics, do away with p-values all together.

---

My answer is condensed and simplified. There are many articles on this topic you can consult for more details, justifications, and specifics, including these:

• Using Effect Size—or Why the P Value Is Not Enough
• Effect sizes and p-values: what should be replicated and what should be reported
• It's the Effect Size, Stupid

Answer 5

"So a laymen like myself expects to not find any p values where there are no hypothesis."

Implicitly, the OP says that in the specific Table he presents, there are no hypotheses that accompany the reported p-values. Just to clear away this small confusion, there certainly are null hypotheses, but they are rather... indirectly mentioned (for economy of space, I presume).

The "p-value" is a conditional probability, say, for a "right-tail" test,

$$\text{p-val} \equiv P(T\geq t(S) \mid H_0) = 1-F_{T|H_0}(t(S) \mid H_0)$$

where $T$ is the statistic used, $F_{T|H_0}(t \mid H_0)$ is the cummulative distribution function that characterizes the probabilities related to $T$ conditional on $H_0$ being true, and $t(S)$ is the value of $T$ obtained by the use of the sample at hand. Obviously, for the test to be meaningful, it must be the case that the statistic $T$ is such and the null hypothesis $H_0$ is such that the distribution of $T$ conditional on $H_0$ being true, is different (or differently parametrized, when they both belong to the same family) from its distribution conditional on $H_0$ not being true.

So a p-value cannot even be calculated if there is no null hypothesis, and whenever we see a p-value reported, somewhere there a null hypothesis lurks.

In the Table presented in the question we read

"All tests for differences across WHR tertiles..."

The null-hypothesis is "hidden" in this phrase: it is "No difference between WHR tertiles", (whatever a "WΗR tertile" is) expressed in its mathematical form which here appears to be a difference of two magnitudes being set equal to zero.

Answer 6

I got curious and read the paper that OP gave as an example: Abdominal obesity increases the risk of hip fracture. I am not a medical researcher and normally do not read medicine papers.

I was surprised to see that the ONLY place where this paper uses $p$-values is the caption of Table 1 that OP reproduced in the question body.

To me it does not look like an "abundance" of $p$-values at all! I am used to neuroscience papers, where different groups of subjects (humans, mice, flies, neurons, tissue samples, etc.) get differently treated or measured in different conditions, and papers usually revolve around the differences between groups. These differences are always assessed with $p$-values, so a paper can have dozens and dozens of them reported in the main text. At times, this really does look like "an abundance". This approach is often (sometimes rightly and sometimes wrongly) criticized for various reasons, see an answer by @Glen_b (+1) and further links.

However, this paper does not do anything like that and only reports $p$-values basically in the introduction, when different characteristics of the cohort are reported. I don't understand what the $p$-values are doing there, and so yes, I agree that they are out of place. However, I don't understand what this whole table is doing there either! I find this table rather confusing (why tertiles? why tertiles of WHR? where is the actual variable of interest, the hip fracture rate?) and it does not seem to be used for any actual analysis further on. This whole table could be kicked out of the text without much loss, together with the $p$-values.

As I do not see any abundance of $p$-values in this paper, I am somewhat confused by the question.

It sounds as if the question is specifically referring to such descriptive tables. If so, this is some weird (but mostly harmless?) practice in medical journals, surviving due to tradition.

---

P.S. By the way, the main analysis of this paper (that does not involve any $p$-values) looks weird to me. The goal of the study is "to examine [...] the relationship between waist circumference (WC), hip circumference (HC), waist/hip ratio (WHR) and BMI to incident hip fracture", while controlling for various possible covariates. Sample size is huge ($n=43000$). What I would do, is to put all predictors into one regression model with an elastic net penalty, select the regularization parameters via cross-validation, and then look at what predictors have non-zero coefficients. Or something similar. The authors, instead, do some ad hoc modeling.

Answer 7

I have to read medical articles often and I feel that the pendulum seems to be swinging from one extreme to another, rather than staying in the central balanced zone.

Following approach seems to work well. If the P value is small, the observed difference is unlikely to be by chance alone. We should, hence, look at the magnitude of the difference and decide whether it is of any practical significance. Very small P values occur with large sample sizes even with very small differences which may be of no practical relevance.

Not including P values in the table of baseline data may be disadvantageous. So if in a study there are two groups with mean ages are 54 and 59 years, I want to know if this difference can be by chance alone. If P is small then I think whether this 5 year difference in 2 groups can affect the results of the study. If P is not small, I do not have to address this question.

Problem occurs if one relies solely on the P value and not check the magnitude of the difference (for example, simple percent change). Some feel that P values should be totally omitted so that only the difference remains and is seen. A balanced solution would be to emphasize on evaluating both these and not to just throw away the P value, which has a limited but 'significant' meaning. The effect size is also likely to correlate closely with P value (just like confidence intervals) and it is also unlikely to completely displace P values from the statistical landscape. As mentioned in following article, there are many virtues of null hypothesis testing because of which it remains popular:

ANTHONY G. GREENWALD, RICHARD GONZALEZ, RICHARD J. HARRIS, AND DONALD GUTHRIE Effect sizes and p values: What should be reported and what should be replicated? Psychophysiology, 33 (1996). 175-183.

Answer 8

The level of statistical peer-review is not as high as one would think from my experience. For all applied papers I have worked on, all of the statistical comments came from experts in the applied field and not from statisticians. For "top" journals, although there is greater scrutiny, it is not uncommon to see results that have serious faults. I think this is partly because the field of statistics can be difficult (as can be seen by disagreements between many of its great minds).

Second, readers in a field expect to see things in a certain way. In one recent experience, I plotted probabilities from a model, but this was shot down because my collaborator guessed correctly this his readers would be more comfortable with a barplot of raw data. In sum, many readers expect to see p-values alongside a table of baseline characteristics.

Unrelated to your direct question, but perhaps relevant: p-values are used in almost every text using frequentist or likelihood methods. The authors often have made tremendous contributions and have thought deeply about statistics. Although abused by experimentalists, surely they have a place in statistics.