Return to Answer

You're confusing probability and $p$ in two parallel ways.

Long run probabilities, like Type I error rates, should not be thought about as directly comparable to the probability that data with values as extreme, or moreso, than the current data are produced by a null model, $p$. And, the $p$ is not the probability of falsely rejecting the null.

Imagine a range of experiments where the null must be true (e.g. comparing two coins for bias). Further imagine selecting various $\alpha$ values prior to running experiments. Won't the smaller $\alpha$'s result in it being less likely that you'll make the Type I error? Would the Type I error be affected at all by the outcome of any one experiment?

I think such confusion often arises because we estimate population parameters while doing testing of a sample. So the mean is an estimate of $\mu$ and the standard deviation is an estimate of $\sigma$ but the $p$ is not a population parameter at all. It's just the probability of the current data or more extreme values if the effect was 0. If you decide that the effect is not 0 then it doesn't mean anything.

You're confusing probability and $p$ in two parallel ways.

Long run probabilities, like Type I error rates, should not be thought about as directly comparable to a conditional probability connected to a single event (the data collected). In this case the latter is the probability that data with values as extreme, or moreso, than the current data are produced by a null model, $p$. And, the $p$ is not the probability of falsely rejecting the null.

Imagine a range of experiments where the null must be true (e.g. comparing two coins for bias). Further imagine selecting various $\alpha$ values prior to running experiments. Won't the smaller $\alpha$'s result in it being less likely that you'll make the Type I error? Would the Type I error be affected at all by the outcome of any one experiment?

I think such confusion often arises because we estimate population parameters while doing testing of a sample. So the mean is an estimate of $\mu$ and the standard deviation is an estimate of $\sigma$ but the $p$ is not a population parameter at all. It's just the probability of the current data or more extreme values if the effect was 0. If you decide that the effect is not 0 then it doesn't mean anything.

You're confusing probability and $p$ in two parallel ways.

Long run probabilities, like Type I error rates, should not be thought about as directly comparable to the probability that data with values as extreme, or moreso, than the current data are produced by a null model, $p$.

The $p$ is not the probability of falsely rejecting the null. Imagine a range of experiments where the null must be true (e.g. comparing two coins for bias). Further imagine selecting various $\alpha$ values prior to running experiments. Won't the smaller $\alpha$'s result in it being less likely that you'll make the Type I error? Would the Type I error be affected at all by the outcome of any one experiment?

I think such confusion often arises because we estimate population parameters while doing testing of a sample. So the mean is an estimate of $\mu$ and the standard deviation is an estimate of $\sigma$ but the $p$ is not a population parameter at all. It's just the probability of the current data or more extreme values if the effect was 0. If you decide that the effect is not 0 then it doesn't mean anything.

You're confusing probability and $p$ in two parallel ways.

Long run probabilities, like Type I error rates, should not be thought about as directly comparable to the probability that data with values as extreme, or moreso, than the current data are produced by a null model, $p$. And, the $p$ is not the probability of falsely rejecting the null. 

Imagine a range of experiments where the null must be true (e.g. comparing two coins for bias). Further imagine selecting various $\alpha$ values prior to running experiments. Won't the smaller $\alpha$'s result in it being less likely that you'll make the Type I error? Would the Type I error be affected at all by the outcome of any one experiment?

I think such confusion often arises because we estimate population parameters while doing testing of a sample. So the mean is an estimate of $\mu$ and the standard deviation is an estimate of $\sigma$ but the $p$ is not a population parameter at all. It's just the probability of the current data or more extreme values if the effect was 0. If you decide that the effect is not 0 then it doesn't mean anything.

You're confusing probability and $p$ in two parallel ways.

Long run probabilities, like Type I error rates, should not be thought about as directly comparable to the probability that data with values as extreme, or moreso, than the current data are produced by a null model, $p$.

The $p$ is not the probability of falsely rejecting the null. Imagine a range of experiments where the null must be true (e.g. comparing two coins for bias). Further imagine selecting various $\alpha$ values prior to running experiments. Won't the smaller $\alpha$'s result in it being less likely that you'll make the Type I error? Would the Type I error be affected at all by the outcome of any one experiment?

I think such confusion often arises because we estimate population parameters while doing testing of a sample. So the mean is an estimate of $\mu$ and the standard deviation is an estimate of $\sigma$ but the $p$ is not a population parameter at all. It's just the probability of the current data or more extreme values if the effect was 0. If you decide that the effect is not 0 then it's doesn't mean anything.

You're confusing probability and $p$ in two parallel ways.

Long run probabilities, like Type I error rates, should not be thought about as directly comparable to the probability that data with values as extreme, or moreso, than the current data are produced by a null model, $p$.

The $p$ is not the probability of falsely rejecting the null. Imagine a range of experiments where the null must be true (e.g. comparing two coins for bias). Further imagine selecting various $\alpha$ values prior to running experiments. Won't the smaller $\alpha$'s result in it being less likely that you'll make the Type I error? Would the Type I error be affected at all by the outcome of any one experiment?

I think such confusion often arises because we estimate population parameters while doing testing of a sample. So the mean is an estimate of $\mu$ and the standard deviation is an estimate of $\sigma$ but the $p$ is not a population parameter at all. It's just the probability of the current data or more extreme values if the effect was 0. If you decide that the effect is not 0 then it doesn't mean anything.