2
$\begingroup$

The p-value is defined as the probability, under the assumption of the null hypothesis H, of obtaining a result equal to or more extreme than what was actually observed.
A p-value lower than a threshold level of significance α means either that the null hypothesis is true and a highly improbable event has occurred, or that the null hypothesis is false.


Does this definition apply to any single measured data point of the sample or only to the "mean" (statistics) of the sample?


I've always thought this methodology is confusing.
This question is not exactly the same as other p-value questions you can find at stackexchange.

Improve this question
$\endgroup$
2
  • $\begingroup$ null hypothesis is true and a highly improbable event has occurred, really? $\endgroup$
    – Hemant Rupani
    Commented Apr 21, 2015 at 12:40
  • 1
    $\begingroup$ @HemantRupani: Yes, really: the quote is referencing Fisher's disjunction. $\endgroup$
    – Scortchi ♦
    Commented Apr 21, 2015 at 13:22

2 Answers 2

Reset to default
9
$\begingroup$

"A result" doesn't mean much out of context—read "test statistic". You choose a test statistic to condense the data into a single number that indexes its discrepancy with the null hypothesis in some direction of interest to you; you calibrate that test statistic by asking how often, from hypothetical repetition of a relevant sampling or random assignment procedure, you'd get values as extreme as, or more extreme than, the one you in fact got.

The value of a single observation $x$, or a function of it,† can be used as a test statistic. Sometimes the same test statistic as you'd use in a larger sample (for a sample size $n$, let $n=1$) will be useful, e.g. the mean or the maximum; sometimes not, e.g. the standard deviation or the range.

† Because more extreme values of $x$ don't always imply greater discrepancy with the null hypothesis.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Then, if I take just a sample of one single observation, What can I say about it? $\endgroup$
    – skan
    Commented Apr 22, 2015 at 2:19
3
$\begingroup$
  • A test statistic is a function that takes all of the data as input and gives you a single number as the output (usually).
  • If the null hypothesis is true, then someone can derive the statistical distribution of the (population) test statistic.
  • If you get a sample test statistic (the test statistic that's calculated using your data) that, according to this known probability distribution, is deemed not very likely, then you reject the null.
  • The p-value is the probability of getting a sample test statistic that is as much or more extreme than the one that you got in real life; the probability is calculated using the known probability distribution of (population) test statistic conditional on the null hypothesis.
  • This methodology makes no sense, yet since everyone uses it, you need to understand it anyway.
Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Note for people new to statistics: The last point qualifies as a statistical joke. Some would want to add that the methodology makes more sense than zero: you should want to worry that you are looking at something that might be a sampling fluke, but opinions differ on how to do that. Others would want to add that they don't do this any way: Bayesians would certainly deny that everyone uses it. $\endgroup$
    – Nick Cox
    Commented Apr 24, 2015 at 15:38
  • 1
    $\begingroup$ This looks a good introductory outline. I would want to quibble that a test statistic is not necessarily a scalar (single number). For example, a plot from one sample could be a test statistic. $\endgroup$
    – Nick Cox
    Commented Apr 24, 2015 at 15:39
  • 1
    $\begingroup$ @Nick, if test statistic is not scalar, then what is the meaning of "more extreme" in the p-value definition? It seems to me that it must take values at least in a partially ordered set... But I don't think I have ever encountered non-scalar test statistic. $\endgroup$
    – amoeba
    Commented Apr 24, 2015 at 20:37
  • 1
    $\begingroup$ For example, a set of quantile-quantile plots could be a sampling distribution. You don't need to quantify a P-value for the idea of "more extreme" to be useful. Several statisticians have used the analogy of a police line-up. If one graph leaps out at you as different, you are on to something. See e.g. Buja et al. rsta.royalsocietypublishing.org/content/367/1906/4361 $\endgroup$
    – Nick Cox
    Commented Apr 24, 2015 at 23:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.