Three questions about the article "Ditch p-values. Use Bootstrap confidence intervals instead"

Question

I am not a statistician by training and I was asked by students to explain them an article called "Ditch p-values. Use Bootstrap confidence intervals instead" . The author seems a prominent academic, however, I am confused about some of the material there. Please, ignore this post if it seems too long for you. I cut it to just 3 questions, I will infer other answers based on these.

Let’s take a simplified but revealing example: we want to determine Robert’s citizenship. Null hypothesis: H0, Robert is a US citizen. Alternative hypothesis: H1, he is not. Our data: we know that Robert is a US senator. There are 100 senators out of 330 million US citizens, so under the null hypothesis, the probability of our data (i.e., the p-value) is 100 / 300,000,000 ≈ 0.000000303. Per the rules of statistical significance, we can safely conclude that our null hypothesis is rejected and Robert is not a US citizen.

Am I right that this is not a p-value (which is the probability to see this or more extreme value of a test statistic)? Is it a correct procedure for a statistical testing? I have a gut feeling that it is a wrong situation to apply hypothesis testing, but I can not formally answer why.

P-values were invented at a time when all calculations had to be done by hand, and so they rely on simplifying statistical assumption. Broadly speaking, they assume that the phenomenon you’re observing obeys some regular statistical distribution

It seems to be wrong, but the question is: can we say that non-parametric tests also rely on some regular statistical distributions? Not only they have assumptions, but also, technically, their statistics also follow some distributions.

Let’s say that a business decision-maker is pondering two possible actions, A and B. Based on observed data, the probability of zero or negative benefits is:

0.08 for action A

0.001 for action B

Should the decision-maker pick action B based on these numbers? What if I told you that the corresponding 90% confidence intervals are:

[-0.5m; 99.5m] for action A [0.1m; 0.2m] for action B Action B may have a lower probability of leading to a zero or negative outcome, but its expected value for the business is much lower, unless the business is incredibly risk-averse.

Can we, based on confidence intervals, say, what is an expected value? Is in this situation a clear decision? I always thought that confidence intervals are not necessarily symmetric, but I started to doubt here.

Answer 1

1 They don’t mean what people think they mean

Am I right that this is not a p-value (which is the probability to see this or more extreme value of a test statistic)? Is it a correct procedure for a statistical testing? I have a gut feeling that it is a wrong situation to apply hypothesis testing, but I can not formally answer why.

One could argue that technically speaking it is a p-value. But, it is a rather meaningless p-value. There are two ways to look at it as a meaningless p-value

• Neyman and Pearson suggest that, in order to compute the p-value, you choose the region where the likelihood ratio (between the null hypothesis and alternative hypothesis) is the highest. You count observations as 'extreme' when a deviation from the null hypothesis would mean more likelihood to make that extreme observation.

  This is not the case with the US citizen example. If the null hypothesis 'Robert is a US citizen' is false, then the observation 'Robert is a US senator' is in no way more likely. So from the viewpoint of Neyman's and Pearson's approach to hypothesis testing, this is a very bad type of calculation for a p-value.

• From the viewpoint of Fisher's approach to hypothesis testing, you have a measurement of some effect and the point of the p-value is to quantify the statistical significance. It is useful as an expression of the precision of an experiment.

  The p-value quantifies how good/accurate the experiment is in the quantification of the deviation. Statistically speaking effects will always occur to some extent due to random fluctuations in the measurements. An observation is seen as statistically significant when it is a fluctuation of a sufficiently large size such that it has a low probability that we observe a seemingly effect when there is actually no effect (when the null hypothesis is true). Experiments that have a high probability that we observe an effect while there is actually no effect are not very useful. We use p-values to express this probability.

  By reporting p-values researchers can show that their experiments have sufficiently small noise and sufficiently large sample size, such that the observed effects are statistically significant (unlikely to be just noise).

  Fisher's p-values are an expression of the noise and random fluctuations, they are a sort of expression of signal/noise ratio. The advice is to only reject a hypothesis when an effect is sufficiently large compared to the noise level.

  Even though there is no alternative hypothesis in Fisher's viewpoint, when we express a p-value then this is done for the measurement of some effect as a deviation relative to a null (no effect) hypothesis. There must be some sense of a direction that can be considered to be an effect or a deviation.

  In the case of the experiment with US citizenship, the measurement of 'Robert is a US senator' has nothing to do with the measurement of some effect or a deviation from the null hypothesis. Expressing a p-value for it is meaningless.

The example with US citizenship may be a bit weird and wrong. However, it is not meant to be correct. The point is to show that simply a p-value is not very meaningful and correct. What we need to consider is also the power of a test (and that is missing in the example with US citizenship). A low p-value might be nice, but what if the p-value would be just as well low, or even lower, for an alternative explanation? If you have a bad hypothesis test then we could 'reject a hypothesis' based on a (crappy) low p-value while actually, no alternative hypothesis is better suitable.

Example 1: Say you have two jars one with 50% gold and 50% silver coins, the other with 75% gold and 25% silver coins. You take 10 coins out of one jar, and they are all silver, which jar do we have? We could say that the prior odds were 1:1 and the posterior odds are 1:1024. We can say that the jar is very likely the one with 50:50 gold:silver, but both hypotheses are unlikely when we observe 10 silver coins and maybe we should mistrust our model.

Example 2: Say you have data that is distributed by a quadratic curve y = a + c x^2. But you fit it with a straight linear line y = a + b x. When we fit a model we find that the p-value is extremely low for a zero slope (no effect) since the data does not match a flat line (as it is following a quadratic curve). But does that mean that we should reject the hypothesis that the coefficient b is zero? The discrepancy, low p-value, is not because the null hypothesis is false, but because out entire model is false (that is the actual conclusion when the p-value is low, the null hypothesis and/or the statistical model is false).

2 They rely on hidden assumptions

It seems to be wrong, but the question is: can we say that non-parametric tests also rely on some regular statistical distributions? Not only they have assumptions, but also, technically, their statistics also follow some distributions

The point of non-parametric tests is that we make no assumptions about the data. But the statistic that we compute may follow some distribution.

Example: We wonder whether one sample is larger than another sample. Let's say that the samples are paired. Then without knowing anything about the distribution we can just count which of the pairs is larger. Independent of the distribution of the population from which the sample has been taken, this sign statistic will follow a binomial distribution.

So the point of non-parametric tests is not that the statistic that is being computed has no distribution, but that the distribution of the statistic is independent from the distribution of the data.

The point of this "They rely on hidden assumptions" is correct. However, it is a bit harsh and sketches the assumptions in a limited sense (as if assumptions are only simplifications to make computations easy).

Indeed many models are simplifications. But I would say that the parametric distributions are still useful, even when we have much more computation power nowadays and simplifications are not necessary. The reason is that parametric distributions are not always simplifications.

• On the one hand: Bootstrapping or other simulations can approach the same result as a computation, and when the computation makes assumptions, approximations and simplifications then the bootstrapping may even do better.

• On the other hand: The parametric distribution, if it is true, gives you information that bootstrapping can't give you. When you have only little amount of data then you can't get a proper estimate of p-values or confidence intervals. With parametric distributions you can fill the gap.

  Example: if you have ten samples from a distribution, then you might estimate the quantile at multiples of 10%, but you won't be able to estimate smaller quantiles. If you know that the distribution can be approximated by some distribution (based on theory and previous knowledge such assumptions might not be bad) then you can use a fit with the parametric distribution to interpolate and extrapolate the ten samples to other quantiles.

  Example 2: The representation of parametric tests as being only useful for making calculations easier is a straw man argument. It is not true because it is far from the only reason. The main reason why people use parametric tests is because they are more powerful. Compare for instance the parametric t-test with the non-parametric Mann-Whitney U test. The choice for the former is not because the computation is easier, but because it can be more powerfull.

3 They detract from the real questions

Can we, based on confidence intervals, say, what is an expected value? Is in this situation a clear decision? I always thought that confidence intervals are not necessarily symmetric, but I started to doubt here.

No, confidence intervals do not give full information. You should instead compute some cost function that quantifies all consideration in the decision (requiring the full distribution).

But confidence intervals may be a reasonable indication. The step from a single point estimate to a range is a big difference and adds an entire new dimension to the representation.

Your criticism here is also exactly the point of the author of the blogpost. You criticize the confidence intervals not giving full information. But the means 0.08 for action A and 0.001 for action B have even less information than the confidence intervals, and that is what the author is pointing out.

This third point is more a matter of point estimate versus interval estimates. Maybe p-values promote the use of point estimates, but it is a bit far-fetched to use it as criticism against p-values. The example is not even a case that is about p-values and it is about a Bayesian posterior for two situations.

Answer 2

"Am I right that this is not a p-value (which is the probability to see this or more extreme value of a test statistic)?" Good question! Yes, you're right, it's not a p-value. What's more the example is not a hypothesis test and it's not a significance test. Anyone who uses it as an argument to discard p-values or hypothesis tests is either confused or attempting to confuse.

The alleged p-value 100 / 300,000,000 is actually the likelihood of a person who is observed to be a US citizen being a senator! Not a p-value and, ironically, calculated on the assumption that the person is a citizen!

The example of US senators is badly adapted from Cohen's original (1994) that was intended to cast doubt on the utility of 'Fisher's disjunction' that underlies the evidential use of p-values.

On Cohen's argument against Fisher's disjunction:

[I've lifted this section from a paper (unpublished: rejected!) that I wrote a decade ago.]

A very highly cited paper that claims Fisher's disjunction is flawed is that of Cohen (1994), who illustrates his claim by drawing an analogy between Fisher's disjunction and this syllogism:

If a person is an American, then he is probably not a member of Congress.

This person is a member of Congress.

Therefore, he is probably not an American.

As Cohen says, the last line of his syllogism about the American is false even though it would be true if the word `probably' were omitted from the first and last lines. However, he is incorrect in suggesting that it is directly analogous to Fisher's disjunction. As Hagen (1997) pointed out in a response published a few years after Cohen's paper, the null hypothesis in Fisher's disjunction refers to the population, whereas in Cohen's syllogism it refers to the sample.

Fisher's disjunction looks like this when put into the form of a syllogism:

• Extreme P-values from random samples are rare under the null hypothesis.

• An extreme P-value has been observed.

• (Therefore, either a rare event has occurred or the null hypothesis
  is false.)

• Therefore, the null hypothesis is probably false.

There is nothing wrong with that, although the line in parentheses is not logically necessary. When Cohen's syllogism is altered to refer to the population, it also is true:

• Members of Congress are rare in the population of Americans.

• This person is a member of Congress.

• (Therefore, either a rare event has occurred or this person is not a random sample from the population of Americans.)

• Therefore, this person is probably not a random sample from the population of Americans.

If a selected person turns out to be a member of Congress then an unusual event has occurred, or the person is a member of a non-American population in which members of Congress are more common, or the selection was not random. Assuming that all members of the American Congress are American there is no relevant non-American population from which the person might have been randomly selected, so the observation casts doubt on the random selection aspect. Cohen is incorrect in his assertion that the Fisher's disjunction lacks logical integrity.

(It is worth noting, parenthetically, that Cohen's paper contains many criticisms of null hypothesis testing that refer to problems arising from the use of what he describes as "mechanical dichotomous decisions around the sacred .05 criterion". He is correct in that, but the criticisms do not directly apply to P-values used as indices of evidence.)

Cohen, J. (1994). The earth is round (p <.05). American Psychologist, 49(12), 997.

Hagen, R. L. (1997). In praise of the null hypothesis statistical test. American Psychologist, 52(1), 15-24.

Answer 3

The author of the article suffers from not understanding that hypothesis tests and confidence intervals serve different inferential purposes:

• The confidence interval (bootstrap or otherwise) serves to provide a plausible range of estimates for a target parameter.

• The hypothesis test serves to make a decision as to whether there is evidence or lack of evidence for a specific claim about a target parameter.

Answer 4

I agree that confidence intervals provide a lot more for performing inference than a single p-value for a single hypothesis, but there is no reason to ditch the p-value and no reason to rely solely on bootstrap confidence intervals. The confidence interval is the set of all hypotheses that are not significant (one would fail to reject) at a specific alpha level. The confidence interval is the inversion of a hypothesis test. If one uses percentiles from a bootstrapped sampling distribution this is a crude approximate confidence interval that does not follow proper construction, but works well nevertheless particularly when the parameter estimator is approximately normally distributed. I find the confidence curve to be a great way to visualize frequentist inference. It shows p-values and confidence intervals of all levels for a parameter of interest, analogous to the Bayesian posterior.

The best solution is to raise awareness and promote continuing education, rather than discarding methods. Here is a link to a great paper by Eric Gibson on the topic. Here is a link to one of my papers regarding confidence curves and visualizing inference.

Gibson, E. (2020). The Role of p-Values in Judging the Strength of Evidence and Realistic Replication Expectations. Statistics in Biopharmaceutical Research. 13(1):1-13

Johnson, G. S. (2021). Decision Making in Drug Development via Inference on Power. Manuscript.

Answer 5

1. Citizenship example

This seems to be a poor but valid test. It makes sense with an extreme p-value cut off $\alpha = 0$. This way we would only reject those citizens whose profession is not found anywhere in the US. So maybe we would say "Robbert cannot be a US citizen because he is a suicide bomber and there are no suicide bombers in the united states" - a valid conclusion. The same in hypothesis testing terms: "$p = 0$ and so the null hypothesis is rejected (obtaining observed sample from H0 is impossible)." Different p-value cut-offs are then just compromise thresholds about how strict we want to be.

2. p-values, calculations by hand, and assumptions

p-values are not tied to computations other than counting the number of occurrences when generating samples for the null hypothesis. And that generation can be performed by physical experiments in the real world. Moreover, we can use p-values for testing non-quantifiable or subjective things. So p-values can stand on their own as a philosophical construction without the need for concrete calculations, and there is no point in stating that they were invented in the age when calculations were performed by hand.

And p-values themselves don't assume anything. It's the statistical tests that have assumptions. And even they are very flexible about that. Sometimes, for example, the assumption is about the sampling distribution of the parameter, and not the observations themselves.

In addition, and I think author acknowledges this in the text, non-parametric tests also make one big assumption - they assume that the observed sample represents all the points from the population. The question is then simply which assumption is more warranted. A simplistic obvious case is with frequencies. Say you want to test if the chance of manufacturing defect is bigger than 5%, you get 10 samples and all of them are without defects. What would you be bootstrapping with 10 zeroes?

3. Confidence intervals

Confidence intervals are a flip-side of the p-value. With p-value you test how surprising your observed sample is given some theoretical null hypothesis and with a confidence interval you define a set (mostly an interval) of hypotheses that would not be surprising given the observed sample. It's somewhat strange to me that so many people criticise one for the other.

4. Other

One different point made against p-values in the text is about business people, to whom the values were reported, didn't understand them. But the answer is simple - don't show people things they might misunderstand. And I really doubt that someone not familiar with the intricacies of a p-value will be able to interpret confidence intervals without mistakes. "The job of a p-value is to prevent you from making a fool of yourself, and not report things as significant" - quoting someone (forgot who, sorry).