3
$\begingroup$

Suppose we sample a person from the population. They are a member of US Congress. We define the null hypothesis $H_0$ as "the person is American". We calculate the $p$-value: $P[member\ of\ Congress | American] \ll 0.05$. Since if the null hypothesis holds, the person is very unlikely to be a member of Congress, we reject the null hypothesis and decide that the person is very likely not an American. This conclusion is obviously very wrong, as all members of Congress are American.

Which assumptions of the hypothesis testing did I violate here? In other words, if I encounter a similar (but more obscure) application where this methodology is also not appropriate, how do I identify it?

Edit: to clarify the methodology. In Fisher's example, where he is trying to identify if the tea-drinking lady can correctly identify cups of tea, he considers the chances of her guessing all cups of tea correctly if she had no ability:

Tea lady

Given that if she had no ability, it would be very unlikely for her to guess all cups of tea correctly, he rejects the hypothesis of her having no ability. So the conclusion is that it is unlikely that she has no ability and just got lucky. Which makes perfect sense.

American

In my example, given that if the person was an American, it would be very unlikely for them to be a member of Congress, we reject the hypothesis of them being an American. So we conclude that it is unlikely that they are an American and we just got lucky picking a Congressman out of all Americans.

So is the reason for this paradox the formulation of the alternative hypothesis? To reach the paradox, we have to assume that the alternative hypothesis forms a disjoint set with the null hypothesis. E.g. alternative of "no ability" is "has ability"; alternative of "population of Americans" is "population of non-Americans". Whereas all we can say is that the alternative hypothesis is some different population than the null hypothesis - it could be a subset! E.g. alternative to "No ability" could be "Someone told her the correct ordering", and alternative to "population of Americans" could be "population of Americans in Washington, DC"? So rejecting the null hypothesis is correct in both cases - it is indeed unlikely that we sampled from all Americans and stumbled on a US congressman - and the paradox arises from jumping to a particular alternative hypothesis and "accepting" it for no good reason? While in fact we cannot say anything about any alternative from the experiment - ALL we know is that the null was unlikely to cause the observation!

|cite|improve this question
$\endgroup$
  • 7
    $\begingroup$ "The person is an American" is not a null hypothesis: it is a prediction about the value of a random variable. It cannot possibly have a p-value. You aren't doing hypothesis testing at all--and since you haven't explained how you obtained your purported "p-value," it isn't evident what you're doing or what your "methodology" might possibly be. Could you edit your post to explain it? $\endgroup$ – whuber♦ Sep 13 '18 at 18:12
  • 1
    $\begingroup$ @whuber "The person is an American" can be a null hypothesis, and the OP is not saying that this has a p-value, they are saying that "The person is a Congressman" does. $\endgroup$ – Acccumulation Sep 13 '18 at 19:28
  • 4
    $\begingroup$ @Acccumulation By definition, a null hypothesis asserts that a population lies within a stipulated set of possible populations. A statement about a randomly selected individual from a population is decidedly not of this type. $\endgroup$ – whuber♦ Sep 13 '18 at 19:36
  • $\begingroup$ The congressperson status or citizenship of a specific person is not even a random variable. $\endgroup$ – random user Sep 18 '18 at 17:25
8
$\begingroup$

Frequentist statistics is meant to make inference on populations using samples, not on individuals. You first define a population (which you have not done), take a sample, and make inference on the population using the sample, taking into account the uncertainty.

You have used your sampled individual as if it were your population, and try to make inference on him. But frequentist statistics do not apply here, you cannot repeat the sampling processs with a population size of 1.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ But is my example not fundamentally the same in this sense as Fisher's lady drinking tea? He had a particular lady, and he wanted to find out if she has "the ability" or not based on how many cups of tea she guessed. So the way I see it, his argument was "If I were to sample from the population of 'no ability' outcomes, a sample with all cups guessed correctly would be very unlikely. Therefore, it is unlikely that the sample I am looking at came from the 'no ability' population". $\endgroup$ – rinspy Sep 13 '18 at 14:34
  • 2
    $\begingroup$ In my case, the argument is "If I were to sample from the population of Americans, a sample that is a member of Congress would be very unlikely. Therefore, it is unlikely that the sample I am looking at came from the 'Americans' distribution". $\endgroup$ – rinspy Sep 13 '18 at 14:35
  • 4
    $\begingroup$ The population in the case of the lady drinking tea is the cups of tea she could possibly drink (which is infinite), not the lady herself. If she does not have the ability, the probability that she guesses right is 0.5. The null hypothesis tested here revolves around this parameter, in your earlier example it revolved around one individual (one teacup by analogy). $\endgroup$ – Knarpie Sep 13 '18 at 14:39
  • 1
    $\begingroup$ The only difference I can see is that in Fisher's example, we implicitly assume that the rare sample is much more likely to have been sampled from some population that is not the null hypothesis population (it doesn't matter which specific population). In my case this assumption does not hold - there is no population that is not the null hypothesis population in which the sample "Member of Congress" is likely. $\endgroup$ – rinspy Sep 13 '18 at 14:40
  • 2
    $\begingroup$ @Knarpie I still do not see the fundamental difference between looking at the probability of drawing a particular sample (e.g. with all 6 cups guessed correctly) from the distribution of random-6-teacup-guess-outcomes, and looking at the probability of drawing a particular sample (e.g. with a person being a US Congressman) from the distribution of being-a-congressman-if-you-are-an-american. Why is the latter not a valid null hypothesis? $\endgroup$ – rinspy Sep 13 '18 at 16:30
6
$\begingroup$

To make this test apply to the population we could change the hypotheses slightly to

H0: The sample is drawn from a population in the US

H1: The sample is drawn from a population not in the US

As far as I can tell there's nothing wrong with this hypothesis test. For a hypothesis test with significance level 0.05 (for example) is a test you need that if the null hypothesis is true then the probability that the test will reject it is less than 0.05.

In this example if you think about repeatedly sampling, if the null hypothesis then we will choose people from the US. And out of those people, only a very small fraction (less than 0.05) are expected to be members of congress, so you would only reject the null hypothesis for less than 5% of them.

So if the test is correct, why does it seem so paradoxical? While it technically satisfies the criterion for a hypothesis test, for any fixed significance level we typically want to choose the rejection criterion which maximizes the power of the test - that is it maximizes the probability of rejecting the null hypothesis if it is false. In your case the test is terrible at this, it will never reject the null hypothesis when it is false.

The paradox depends on the rejection criterion being impossible under the alternative hypothesis or more unlikely than under the alternate hypothesis that under the null. Any such test will have zero or very low power.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Intuitively, the assumption that "the rejection criterion must be (much) more likely under the alternate hypothesis than under the null" seems key. In fact, the more I think about it, the more it seems like an implicit assumption in this kind of statistical testing. $\endgroup$ – rinspy Sep 13 '18 at 14:17
  • $\begingroup$ Agreed. I think it's so implicit because it's usually not a problem, the null hypothesis is typically quite specific and the alternative quite broad (e.g. a specific value of a parameter vs all others or independence vs dependence), but it's still probably not mentioned enough. $\endgroup$ – Rohan Sep 13 '18 at 15:40
  • 1
    $\begingroup$ I've altered my answer slightly to take into account the other answer which I think raises a valid point. But the idea hasn't changed. $\endgroup$ – Rohan Sep 13 '18 at 16:03
  • $\begingroup$ See the edit on my Q: I think the fundamental problem that leads to the paradox is the formulation of $H_1$ the way you stated it. The correct $H_1$ is "The sample is not drawn from the US general population", which is subtly different from yours. This still allows for the sample to be drawn from Americans in Washington DC, for example. And then there is no paradox. Basically, the alternative hypothesis should cover ANY possible distribution that is not exactly the null distribution. $\endgroup$ – rinspy Sep 14 '18 at 10:34
  • $\begingroup$ "the alternative hypothesis should cover ANY possible distribution that is not exactly the null distribution." This depends on what the other possibilities are - in my example you could imagine that you are explicitly told that your population is the entire population of a country. Obviously that's a pretty contrived example but I think the point remains, if you want a more realistic one then think about one-tailed vs two-tailed tests. $\endgroup$ – Rohan Sep 16 '18 at 18:09
4
$\begingroup$

I'd say there's at least two additional problems with your "paradox" (I don't even think it is a valid testing problem):

First, "Member of Congress=1" I'd say a useless test statistic $T(X)$ for your "H0" as it does not measure deviation from the H0. So, a person who is not American would have automatically "Member of Congress=0" which also applies to most Americans. A test statistic should be something that collects evidence against the H0. If enough evidence is collected, we eventually reject H0. You need a point that is in in perfect accordance with the H0, say T(X)=0; then a T(X)≠0 should carry more information against H0 then T(X)=0. Let me expand on that. What values can the test statistic take? Well 1 if the person is a member of congress AND American and 0 if the person is either American AND not member of congress, OR Non-American. That means that the test statistic can take on TWO distinct values (0, 1) if the null were true! And both values do carry information in favour of the H0 (0 for Americans non-congress members, 1 for American congress members). But one of these values (0) also carries information in favour against the H0. So what does one learn about H0 in case the test statistic is 0 or not 0, especially with n=1? Thus the test you describe appears invalid to me (in the sense of the test keeping its $\alpha$ for every $\alpha \in [0;1]$). It is most certainly also inadmissible, i.e., there is a test for that H0 having better power.

Second, the p-value is defined as the probability to observe a test statistic that speaks as much or more strongly against the H0 as the value you have observed in the sample. In other words, it is a quantile of the distribution of the test statistic under the assumption the null were true. I have difficulties to match your p-value, which seems to simply be a conditional probability, to that. But not every conditional probability conditioning on "American" is automatically the correct p-value because for that one would have to work out the correct null distribution of the test statistic that you propose. Here $T(X)$ is 1 iff congress & American. 0 if American & not congress or 0 if Non-American, for which I have difficulties translating what you call "H0" properly to a condition under which to derive the null distribution. Would you say the value of $T(X)$ corresponding to perfect agreement with H0 is $T(X)=1$ or $T(X)=0$, or both, for the population of "all people on earth"? Or do you restrict yourself to "all people in the US", "people who are US citizens" etc.? All that would be needed to derive the proper null distribution for the test statistic.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ If I understand you correctly, by "it does not measure deviation from the $H_0$" you mean that we should also consider the distribution of the test statistic under the alternative hypothesis? With regards to the correct null distribution of the test statistic - is it not just the distribution of "members of congress" among "americans"? $\endgroup$ – rinspy Sep 14 '18 at 9:29
  • $\begingroup$ First Q: "should we also consider the distribution of the test statistic under the alternative hypothesis": Yes we should (see other answers), but that is tangential to what I meant here, which was that a test statistic $T(X)$ should be something that collects evidence against the H0. If enough evidence is collected, we eventually reject H0. You need a point that is in in perfect accordance with the H0, say $T(X)=0$; then a $T(X)\neq 0$ should carry more information against H0 then $T(X)=0$. $\endgroup$ – Momo Sep 14 '18 at 10:11
  • $\begingroup$ See the edit on my Q. I think I understand it now. We are right in rejecting the null hypothesis - it is in fact unlikely that the congressman was randomly sampled from all Americans, because congressmen are rare among all Americans! The paradox arises from incorrectly "accepting" a specific alternative hypothesis. The correct alternative is that we did not sample the person randomly from all Americans - no more, no less. That could mean that we instead sampled the person from all Americans in Washington, DC, for example. $\endgroup$ – rinspy Sep 14 '18 at 10:15
  • $\begingroup$ Second Q: Not automatically. Your $T(X)$ is 1 iff congress & American. 0 if American & not congress or 0 if Non-American. Ok. But I have difficulties translating what you call "H0" properly to a condition under which to derive the null distribution. $\endgroup$ – Momo Sep 14 '18 at 10:18
0
$\begingroup$

When we do hypothesis testing, we need to decide how willing we are to be wrong. $\alpha$ is the proportion of times when $H_o$ is true that we reject it. When we reject the null hypothesis, we are not saying that the null hypothesis definitely is false, we're saying that we have performed a test that has at most $\alpha$ chance of being wrong. If you were to try the hypothesis testing you describe, you would have less than 5% change of incorrectly rejecting the null hypothesis.

Note that your formulation is ordered incorrectly, however. In a valid hypothesis test, you have to state what the rejection region is before collecting any statistics. You describe observing whether the person is a member of Congress before you describe this being part of the rejection region. I'm taking your order of description to be chronological order; if you were to decide after seeing that the person is a member of Congress that that should be a basis of rejection $H_o$, that would be post hoc hypothesis testing, which isn't valid.

It is not correct phrasing to saying that the null hypothesis is unlikely to true. In hypothesis testing, we are decisioning entirely on the probability of seeing the result given the null hypothesis, not the probability of the null hypothesis given the result. "we ... decide that the person is very likely not an American" would be fallacious even if you had a better test.

It absolutely true that in doing hypothesis testing, there is some sense in which it is not necessary to pick a statistic that makes sense. If you roll a D20 and reject the null hypothesis whenever it comes up 20, that is a hypothesis test with $\alpha=.05$. Your example shows that statistical tests should be evaluated based not only on $\alpha$, but on power (1-$\beta$). In my example, the power is 5%: if the null hypothesis is false, I have only 5% chance of rejecting it. Since the power and $\alpha$ are equal, this test doesn't actually have any diagnostic value. Whether a hypothesis test is valid, and whether it is at all worthwhile, are two different issues. You can't tell just from the p-value how much you should update your beliefs.

In you example, the probability of a non-American being a member of Congress is zero. So your power is less than your $\alpha$, which makes your test worse than useless.

|cite|improve this answer
$\endgroup$
0
$\begingroup$

I can't say for certain, but I'm guessing you're making a rather frequent mistake:

You're (implicitly) misinterpreting p-values.

A very, very common misinterpretation of p-values is "a p-value is the probability that the null hypothesis is correct." That's wrong. Instead, the p-value is the subtly different probability that, given that the null hypothesis is correct, what's the probability that you'll see results as or more extreme than you did?

This misintepretation is somewhat implicit in your logic. I'm guessing you're thinking along the lines of "Okay, the p-value says the null hypothesis is really, really unlikely. Wait - what gives? The setup of the experiment is such that it should be very, very obvious that the null hypothesis is near certain? Why is the p-value near zero when the probability of the null hypothesis being true is near one?" -- The answer to that is, again, that the p-value is not measuring the probability that the null hypothesis is true.

So what is the p-value? It's not the probability of the null hypothesis (the p-value conditioned on the null hypothesis being true, so certainly it can't be that). Instead it's the probability of the data. -- But remember we're in a frequentist framework. What does "the probability of the data" actually mean? For frequentists, the probability is the number of times an event happens over the total number of trials. The data we have is the data we have. Number of trials equals one, so how are we getting fractions here?

The trick is that the number of trials is not our actual experiment. Instead, it's a hypothetical set of experiments under the assumption the null hypothesis is true. We don't run the (theoretical) experiment once, we run it multiple times. If we do this (sample under the assumption that the people we sample are Americans), we see that the chance of getting "congressman-ness" at-or-more-extreme than our initial trial is very, very low. In fact, it's equal to the p-value.

Implicit in the concept of using p-values in null hypothesis testing is the concept that you're sometimes going to get it wrong. You will occasionally get experimental results where you'll inappropriately discard the null hypothesis. In fact, you'll inappropriately discard it (p-value) fraction of the time that the null hypothesis is actually true. (Assuming a large number of independent experiments.)

So, yeah, in your example the null hypothesis is true and you were unlucky and happened to get results which inappropriately discarded it. But the p-value tells you there's a small probaility that would happen. If you repeated the experiment (given that the null hypothesis is true), what's the chance you'll get a congressman the next time? Pretty darn low, just like the p-value says.

Also, 0.05 is not a magic number. It's rather arbitrary. There isn't much a priori rationale in choosing 0.05. R.A. Fisher used it, so it has a certain cachet, but it's not the most appropriate value in all situations. One in 20 just seemed like a reasonable compromise at the time between falsely rejecting the null hypothesis versus making things too stringent and never rejecting the null hypothesis, even when you should.

In particular, if you're concerned about making Type I errors (that is, rejecting the null hypothesis when it's actually true), you will need to tighten your p-value cutoff. The most common example of this is with multiple-testing situations, where you're performing a number of null hypothesis tests of different sorts. You will get a certain error rate, so people reduce the p-value threshold to avoid spurious rejections. But you also can (and should) change your p-value cutoff if the damage of incorrectly rejecting the null hypothesis is significantly greater than missing situations where the null hypothesis is incorrect.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I agree with everything you have written, but I think it still does not address the paradox. The problem here is that every time we (rarely) sample a US Congressman, we will be wrong to reject the null hypothesis every time! But I do think the paradox stems from misinterpretation and some implicit assumptions built into this kind of hypothesis testing. The fundamental problem is that just because your sample is very unlikely under the null hypothesis does not mean that it is more likely under some alternative hypothesis, and so "rejecting" the null in general is incorrect. $\endgroup$ – rinspy Sep 14 '18 at 8:25
  • $\begingroup$ All you can say from $P[observation|H_0]$ being very low is that you would be very unlikely to observe the observation if $H_0$ were true. That is all. To say anything about the likelihood of $H_0$ being true given the observation, you must do Bayesian inference. $\endgroup$ – rinspy Sep 14 '18 at 8:27
  • $\begingroup$ @rinspy And every time you get struck by lightening, you get struck by lightning. Frequentist null hypothesis testing doesn't claim to be perfect -- it's explicitly built around the concept of getting it wrong! Yes, you wrongly reject the null hypothesis some small fraction of the time, but if you're doing it right, that happens rarely. ... In fact, if you're frequently picking congressmen, you're completely justified in rejecting the null hypothesis (that you're picking Americans at random). $\endgroup$ – R.M. Sep 14 '18 at 14:25
  • $\begingroup$ And again, null hypothesis testing doesn't say anything about the probability of the null hypothesis being true. It doesn't claim to, and it's a mistake to say that it does. It also doesn't tell you anything about the likelihoods of alternative hypothesis. Yes, there's a subjective component to picking the p-value threshold, but with the Bayesian approach there's a subjective component to picking the priors. $\endgroup$ – R.M. Sep 14 '18 at 14:28
0
$\begingroup$

Your definition of p-value is not right here. Note that p-value is the probability under null to get at least as extreme observation as the observed one. You defined it as the probability of getting the observation under null. Let's formalize the problem. Essentially you are drawing a sample of size 1 from an unknown population, and you want to test whether the population proportion, $\pi$, of Americans in your population is equal to $1$. Your alternative hypothesis is that $\pi=0$.

In your sample, you have $X$ = Number of congressmen in the sample, which can be either 0 or 1. To define p-value, you have to ask "Which value of $X$ is a larger evidence against the null?" Clearly it is 0. Therefore your p-value should be defined as $P_{H_0}(X \leq X_{obs})$, which is not small when $X=1$. In fact, it is equal to 1. Even if you get $X=0$ in your sample, your p-value will still not be small and you will end up accepting the null either way. That is because you have a very low-powered test due to having sample size = $1$.

Again, focus on the part 'at least as extreme as the observed one'. In the lady tasting tea problem, the lady getting correct answer is more evidence against the null, therefore the higher the number of correct answers the more extreme it is. Since she got all of them correct in the sample, the observed data was the most extreme situation. Therefore the p-value was defined as the probability of that case only. If she answered only one wrong, p-value would have been probability of getting one wrong + probability of getting correct.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ 'Clearly it is 0' Why? $\endgroup$ – innisfree Aug 15 at 7:31
  • $\begingroup$ @innisfree Because X = 1 automatically implies null, whereas X = 0 may be from either the null or the alternative. Therefore, X = 0 provides more evidence against null compared to X = 1. $\endgroup$ – Pratyay Aug 16 at 14:29
  • $\begingroup$ I think it depends on the alternative, though. If the alternative were ‘population of US congress members’, X=1 would be more extreme, right? $\endgroup$ – innisfree Aug 18 at 7:54
  • $\begingroup$ The alternative cannot be ‘population of US congress members’ since the null is "the person is American". Null and alternative cannot overlap. I believe the question implies that the alternative is "the person is not American". But even if it doesn't, an alternative such as "the person is British" would still lead to the same argument that I wrote in my answer. $\endgroup$ – Pratyay Aug 19 at 13:16
  • $\begingroup$ !? We have a data sample. We wish to consider H1: samples drawn from population of Congress members and H0: samples drawn from US population. Are you really saying we can’t use hypothesis testing here? $\endgroup$ – innisfree Aug 19 at 15:16
-2
$\begingroup$

Your null hypothesis seems to be "This person is American"

So presumably your alternative hypothesis is "This person is not American"

  • "This person is a member of the US Congress" is then not an extreme result in terms of being American or not. Indeed it points towards the null hypothesis

  • "This person is a king or queen or prince or princess" might be an extreme result in terms of the hypotheses, even taking Megan Markle and Lisa Halaby (and a few others) into account

|cite|improve this answer
$\endgroup$

protected by whuber♦ Sep 18 '18 at 17:51

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.