15
$\begingroup$

So, this may be a common question, but I’ve never found a satisfactory answer.

How do you determine the probability that the null hypothesis is true (or false)?

Let’s say you give students two different versions of a test and want to see if the versions were equivalent. You perform a t-Test and it gives a p-value of .02. What a nice p-value! That must mean it’s unlikely that the tests are equivalent, right? No. Unfortunately, it appears that P(results|null) doesn’t tell you P(null|results). The normal thing to do is to reject the null hypothesis when we encounter a low p-value, but how do we know that we are not rejecting a null hypothesis that is very likely true? To give a silly example, I can design a test for ebola with a false positive rate of .02: put 50 balls in a bucket and write “ebola” on one. If I test someone with this and they pick the “ebola” ball, the p-value (P(picking the ball|they don’t have ebola)) is .02, but I definitely shouldn’t reject the null hypothesis that they are ebola-free.

Things I’ve considered so far:

  1. Assuming P(null|results)~=P(results|null) – clearly false for some important applications.
  2. Accept or reject hypothesis without knowing P(null|results) – Why are we accepting or rejecting them then? Isn’t the whole point that we reject what we think is LIKELY false and accept what is LIKELY true?
  3. Use Bayes’ Theorem – But how do you get your priors? Don’t you end up back in the same place trying to determine them experimentally? And picking them a priori seems very arbitrary.
  4. I found a very similar question here: stats.stackexchange.com/questions/231580/. The one answer here seems to basically say that it doesn't make sense to ask about the probability of a null hypothesis being true since that's a Bayesian question. Maybe I'm a Bayesian at heart, but I can't imagine not asking that question. In fact, it seems that the most common misunderstanding of p-values is that they are the probability of a true null hypothesis. If you really can't ask this question as a frequentist, then my main question is #3: how do you get your priors without getting stuck in a loop?

Edit: Thank you for all the thoughtful replies. I want to address a couple common themes.

  1. Definition of probability: I'm sure there is a lot of literature on this, but my naive conception is something like "the belief that a perfectly rational being would have given the information" or "the betting odds that would maximize profit if the situation was repeated and unknowns were allowed to vary".
  2. Can we ever know P(H0|results)? Certainly, this seems to be a tough question. I believe though, that every probability is theoretically knowable, since probability is always conditional on the given information. Every event will either happen or not happen, so probability doesn't exist with full information. It only exists when there is insufficient information, so it should be knowable. For example, if I am told that someone has a coin and asked the probability of heads, I would say 50%. It may happen that the coin is weighted 70% to heads, but I wasn't given that information, so the probability WAS 50% for the info I had, just as even though it happens to land on tails, the probability WAS 70% heads when I learned that. Since probability is always conditional on a set of (insufficient) data, one can never not have enough data to determine it and so it should always be (theoretically) knowable.
    Edit: "Always" may be a little too strong. There may be some philosophical questions for which we can't determine probability. Still, in real-world situations, while we can "almost never" have absolute certainty, there should "almost always" be a best estimate.
Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ If your 'null hypothesis' is something like $H_{0}: \theta = 0$, that is, that some difference is zero, then rejecting it means you have found strong enough evidence that $H_{A}: \theta = 0$. You could instead for a null hypothesis like $H_{0}: |\theta| \ge \Delta$, that is, that some difference is at least as big as $\Delta$ (where $\Delta$ is what the researcher deems the smallest difference they care about), and rejecting means that you found $H_{A}: |\theta| < \Delta$ (i.e. $-\Delta < \theta < \Delta$). See tests for equivalence stats.stackexchange.com/tags/tost/info $\endgroup$
    – Alexis
    Commented Sep 27, 2017 at 21:01
  • $\begingroup$ The power of an experiment (and of the statistical test analyzing the experiment's outcomes) is the probability that if there were an effect of a given size or larger, that the experiment wouold detect it at a given threshold of significance. statisticsdonewrong.com/power.html $\endgroup$
    – Bennett Brown
    Commented Sep 28, 2017 at 6:13
  • $\begingroup$ see stats.stackexchange.com/questions/166323/… $\endgroup$
    – user83346
    Commented Sep 28, 2017 at 12:58
  • $\begingroup$ Your coin example is a good one. It shows that you can never know P(H0|results) if you only know the results and make no further assumptions. Do you know the probability of heads in a given throw 'assuming' a certain fairness of the coin? Yes. (but this is hypothetical, given the assumptions, and you will never know if your assumptions are true) Do you know the probability of heads in a given throw while knowing a number of previous outcomes. No! and it does not matter how large the number of previous outcomes you know. You can not exactly know the probability heads in the next throw. $\endgroup$
    – Sextus Empiricus
    Commented Sep 28, 2017 at 15:02

4 Answers 4

Reset to default
14
$\begingroup$

You have certainly identified an important problem and Bayesianism is one attempt at solving it. You can choose an uninformative prior if you wish. I will let others fill in more about the Bayes approach.

However, in the vast majority of circumstances, you know the null is false in the population, you just don't know how big the effect is. For example, if you make up a totally ludicrous hypothesis - e.g. that a person's weight is related to whether their SSN is odd or even - and you somehow manage to get accurate information from the entire population, the two means will not be exactly equal. They will (probably) differ by some insignificant amount, but they won't match exactly. ' If you go this route, you will deemphasize p values and significance tests and spend more time looking at the estimate of effect size and its accuracy. So, if you have a very large sample, you might find that people with odd SSN weigh 0.001 pounds more than people with even SSN, and that the standard error for this estimate is 0.000001 pounds, so p < 0.05 but no one should care.

Improve this answer
$\endgroup$
18
  • 1
    $\begingroup$ Not that I disagree with you, but don't you think when he worries about p(data|H0) or p(H0|data) he's talking about studies with low $n$. The example you give is easy in both frameworks bayesian and frequentist because their respective weaknesses/subjectivity don't matter in light of abundant data. The only error you can still make in this situation that would matter is to confuse significance with effect size. $\endgroup$
    – David Ernst
    Commented Sep 27, 2017 at 20:45
  • 1
    $\begingroup$ Good point about effect size. Is there an analogue to situations like testing for a disease, where the question is Boolean in nature? $\endgroup$
    – Kalev Maricq
    Commented Sep 28, 2017 at 13:57
  • 1
    $\begingroup$ FWIW, I'm perfectly willing to believe that there is no relationship between a person's weight & whether their SSN is odd or even. In an observational study, these variables will be correlated w/ some other variables, etc, such that there is ultimately a non-0 marginal association. I think the valid point is that, for most things researchers invest their time to investigate, there is some decent reason to suspect that there is real a non-0 effect. $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 28, 2017 at 14:53
  • 1
    $\begingroup$ @gung you can believe whatever you want, but there is definitely a non-zero relationship between weight and SSN. We do know anything more about the relationship other than its existence and that it is probably small. $\endgroup$
    – emory
    Commented Sep 28, 2017 at 15:34
  • 1
    $\begingroup$ I know that weight is a continuous variable. Although we might record it as an integer number of kilograms. Your comment was about an observational study (drawing inferences about a population based on a sample). Since my study is funded by hypothetical dollars it is a population study using infinite precision scales - no need for statistical inference. $\endgroup$
    – emory
    Commented Sep 28, 2017 at 15:57
3
$\begingroup$

In order to answer this question, you need to define probability. This is because the null hypothesis is either true (except that it almost never is when you consider point null hypotheses) or false. One definition is that my probability describes my personal belief about how likely it is that my data arose from that hypothesis compared to how likely it is that my data arose from the other hypotheses I'm considering. If you start from this framework, then your prior is merely your belief based on all your previous information but excluding the data at hand.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Good point. I think my idea of probability is something like "the perfectly rational belief" instead of my personal one. I edited my question to address your points. $\endgroup$
    – Kalev Maricq
    Commented Sep 28, 2017 at 14:07
2
$\begingroup$

The key idea is that, loosely speaking, you can empirically show something is false (just provide a counterexample), but you cannot show that something is definitely true (you would need to test "everything" to show there are no counterexamples).

Falsifiability is the basis of the scientific method: you assume a theory is correct, and you compare its predictions to what you observe in the real world (e.g. Netwon's gravitational theory was believed to be "true", until it was found out that it did not work too well in extreme circumstances).

This is also what happens in hypothesis testing: when P(results|null) is low, the data is contradicting the theory (or you were unlucky), so it makes sense to reject the null hypothesis. In fact, suppose null is true, then P(null)=P(null|results)=1, so the only way that P(results|null) is low is that P(results) is low (tough luck).

On the other hand, when P(results|null) is high, who knows. Maybe null is false, but P(result) is high, in which case you cannot really do anything, besides designing a better experiment.

Let me reiterate: you can only show that the null hypothesis is (likely) false. So I would say the answer is half of your second point: you don't need to know P(null|results) when P(results|null) is low in order to reject null, but you cannot say null is true it P(results|null) is high.

This is also why reproducibility is very important: it would be suspicious to be unlucky five times out of five.

Improve this answer
$\endgroup$
3
  • $\begingroup$ "you can empirically show something is false" I believe that rejection of a hypothesis is similarly problematic as acception. A p-value is not equal to the probability that the null hypothesis is false. Otherwise, in the sense of Alexis' comment on the OP we could define $H_0:$ |result|>a. And proof it to be false by finding counter examples when observing result<a. Thus showing $H_{alternative}:$ |result|<a is true. $\endgroup$
    – Sextus Empiricus
    Commented Sep 28, 2017 at 12:24
  • $\begingroup$ I agree with Martijn. If you can tell me how to determine the probability that the null hypothesis is false, I would consider that a successful answer to my question. $\endgroup$
    – Kalev Maricq
    Commented Sep 28, 2017 at 14:17
  • $\begingroup$ also note that P(result|null) being small can be normal even if the null is true. For instance if we observe the average in 1000 dice rolls, $\mu_{1000}$, then $P(\mu_{1000}=3.50)$ is small even for a fair dice. p-values are constructed differently than P(result|null), and are more precisely made to define the type I error, by describing 'result' as 'the result at which we reject'. In that way we have type I error as P(null rejected | null true) = P(rejection result|null). So imagine the null is true (hypothetically) then we have P(rejection result|null) probability to make a type I error. $\endgroup$
    – Sextus Empiricus
    Commented Sep 28, 2017 at 19:45
2
$\begingroup$

-----------------------------------------------------------------------

(edit: I think it would be useful to put a version of my comment to this question on top in this answer, as it is much shorter)

The non symmetric computation of p(a|b) occurs when it is seen as a causal relationship, like p(result|hypothesis). This computation does not work in both directions: a hypothesis causes a distribution of possible results, but a result does not cause a distribution of hypotheses.

P(result|hypothesis) is theoretic value based on the causation relationship hypothesis -> result.

If p(a|b) expresses a correlation, or observed frequency (not necessarily a causal relation), then it becomes symmetric. For instance if we write down the number of games a sports teams wins/loses and number of games sports team scores less than or equal as / more than 2 goals in a contingency table. Then P(win|score>2) and P(score>2|win) are similar experimental/observational (not theoretic) objects.

---------------------------------------------------------------------

Very simplistic

The expression P(result|hypothesis) seems so simple that it makes one easily think that you can simply reverse the terms. However, 'result' is a stochastic variable, with a probability distribution (given the hypothesis). And 'hypothesis' is not (typically) a stochastic variable. If we make 'hypothesis' a stochastic variable then it implies a probability distribution of different possible hypothesises, in the same way as we have a probability distribution of different results. (but results does not give us this probability distribution of hypothesis, and merely changes the distribution, by means of Bayes theorem)

An example

Say you have a vase with red/blue marbles in a 50/50 ratio from which you draw 10 marbles. Then you can easily express something like P(outcome|vase experiment), but it makes little sense to to express P(vase experiment|outcome). Outcome is (on it's own) not the probability distribution of different possible vase experiments.

If you have multiple possible types of vase experiments, in that case it is possible to use express something like P(type of vase experiment) and use Bayes rule to get a P(type of vase experiment|outcome), because now the type of vase experiment is a stochastic variable. (note: more precisely it is P(type of vase experiment|outcome & distribution of type of vase experiments))

Still, this P(type of vase experiment|outcome) requires a (meta-)hypothesis about a given initial distribution P(type of vase experiment).

Intuition

maybe the expression below helps to understand the one direction

X) We can express the probability of X given a hypothesis about X.

thus

1) We can express the probability for results given a hypothesis about the results.

and

2) We can express the probability of a hypothesis given a (meta-)hypothesis about these hypothesises.

It is Bayes rule that allows us to express an inverse of (1) but we need (2) for this, hypothesis needs to be a stochastic variable.

Rejection as solution

So we can not obtain a absolute probability for a hypothesis given the results. That is a fact of life, trying to fight this fact seems to be the origin of not finding a satisfactory answer. The solution to find a satisfactory answer is: accepting that you can not get a (absolute) probability for a hypothesis.

Frequentists

In the same way as not being able to accept a hypothesis, we should neither (automatically) reject the hypothesis when P(result|hypothesis) is close to zero. It only means that there is evidence that supports change of our believes and it depends also on P(result) and P(hypothesis) how we should express our new believes.

When frequentists have some rejection scheme then that is fine. What they express is not wether a hypothesis is true or false, or the probability for such cases. They are not able to do that (without priors). What they express instead is something about the failure rate (confidence) of their method (given certain assumptions are true).

Omniscient

One way to get out all of this is to elliminate the concept of probability. If you observe the entire population of 100 marbles in the vase then you can express certain statements about a hypothesis. So, if you become omniscient and the concept of probability is irrelevant, then you can state wether a hypothesis is true or not (although probability is also out of the equation)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Your vase example makes sense. However, in real life, we almost never know how many marbles of each color are in the vase. I always find myself with a question more like "Are there more red marbles than blue" and my data is that I drew 4 red marbles and 1 blue marble from the vase. Now, I can make assumptions like "there are probably ~100 marbles and each marble is either red or blue with 50% probability" but in real life, I often find myself at a loss for how to non-arbitrarily and non-circularly get these priors. $\endgroup$
    – Kalev Maricq
    Commented Sep 28, 2017 at 14:04
  • $\begingroup$ That is more an epistemological question than a problem about probability. An expression like P(result|hypothesis) is in a similar way "false", I mean, it is a hypothetical expression. You can express the probability for a result, given a certain hypothetical believe about 'reality'. In the same way as a probability for an experimental outcome is hypothetical, an expression for the probability of some theory (with or without some observation of a result), requires a certain hypothetical believe about 'reality'. Yes, priors are somewhat arbitrary. But so is a hypothesis. $\endgroup$
    – Sextus Empiricus
    Commented Sep 28, 2017 at 14:34
  • $\begingroup$ Talking about the probabilities. Note that Bayes rule is about two stochastic variables: P(a|b) P(b) = P(b|a) P(a). You can relate the conditional probabilities. If one of those P(b|a) is a causal relationship, as in 'theory leads to distribution of outcomes', then you can calculate it exact. Such case is only because the (1directional) causality. The hypothesis allows to know (hypothetic) everything you need, the marbles in the vase. The other way around, does not work. An experimental outcome 4 red vs 1 blue, does not cause the probability distribution of marbles in the vase. $\endgroup$
    – Sextus Empiricus
    Commented Sep 28, 2017 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.