1
$\begingroup$

This question already has an answer here:

We know that a p-value can be interpreted as the probability of seeing the same response given that the null hypothesis is true.

p-value is not the probability of making a type 1 error. (as emphasized here: http://blog.minitab.com/blog/adventures-in-statistics/how-to-correctly-interpret-p-values)

However, the significance level, alpha, we know corresponds to the probability type 1 error.

How does it make sense to make comparison with alpha and p-value? They represent to different probabilities.

$\endgroup$

marked as duplicate by Scortchi Mar 28 '16 at 16:29

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Your opening sentence is wrong, or too sloppy for use here. Neither is it standard to equate "the significance level" with "alpha". Perhaps you should edit your question, but I suspect that it has been answered already several times before. Use the search function for P-values and alpha. $\endgroup$ – Michael Lew Feb 24 '16 at 3:48
  • 1
    $\begingroup$ Welcome to CV unknown! While I agree with @MichaelLew regarding the definitions, I think you are quite welcome to ask this question here. After all, it is not wrong to be incorrect, and we learn by making mistakes. Hopefully I or someone else will provide an answer that helps clarify. $\endgroup$ – Alexis Feb 24 '16 at 3:55
  • 1
    $\begingroup$ I found that definition online. I felt it was important to ensure that we were on the same page. $\endgroup$ – user46925 Feb 24 '16 at 3:56
  • 1
    $\begingroup$ Yeah, my comment now looks more grumpy than I actually felt. Sorry. $\endgroup$ – Michael Lew Feb 24 '16 at 4:02
3
$\begingroup$

a p-value can be interpreted as the probability of seeing the same response given that the null hypothesis is true.

Not quite: the p-value is the probability of observing a test statistic that is as or more extreme than the one you actually observed. This is why the p-value is expressed as a probability in terms of an inequality: $p = P\left(\Theta \ge \theta\right)$, $p = P\left(\Theta \le \theta\right)$, or $p = P\left(|\Theta| \ge |\theta|\right)$.

However, the significance level, alpha, we know corresponds to the probability type 1 error.

Almost: $\alpha$ is a researcher choice, and represents the willingness of the researcher to falsely reject a single null hypothesis, assuming that the null hypothesis is true.

In the frequentist world it makes sense to reject test statistics that are too unlikely... too extreme assuming the null hypothesis is true. The reasoning is along the lines of "if this test statistic, $\theta$, was so unlikely to be observed if the null hypothesis is true, then perhaps the null hypothesis isn't true!" The p-value is the measure of "how extreme" under the null.

The value of $\alpha$ indicates the researcher's line in the sand: probabilities of observing test statistics as or more extreme as $\alpha$ (because they are smaller than $\alpha$) are rejected because both probabilities are defined under the assumption that the null hypothesis—including the distribution implied by it—is true.

$\endgroup$
  • $\begingroup$ Hmm, what does extreme mean? If I understand, the p-value is the probability that we'll see a response as powerful as we saw the sample data? $\endgroup$ – user46925 Feb 24 '16 at 4:02
  • 3
    $\begingroup$ A more extreme test statistic value is a value of that statistic that can be expected to occur less commonly according to the relevant sampling distribution. In most cases a more extreme value of the test statistic has a larger absolute value, and corresponds to a dataset that can be thought of as more surprising according to the null hypothesis. $\endgroup$ – Michael Lew Feb 24 '16 at 4:06
  • $\begingroup$ @unknown "More extreme" is right there in the $p = P(\Theta \ge \theta)$, $p = P(\Theta \le \theta)$, and $p = P(|\Theta| \ge |\theta|)$. Notice that none of those expressions are $p = P(\Theta =\theta)$: the inequality ($\ge$ or $\le$) means as or more extreme. $\endgroup$ – Alexis Feb 24 '16 at 19:06
  • $\begingroup$ @MichaelLew +1 for the "in most cases." :) $\endgroup$ – Alexis Feb 24 '16 at 19:07