P-Value Clarification [duplicate]

Question

I'm new to P-Values and Hypothesis testing and I noticed that p-values are explained differently everywhere I go. In my lecture notes it has a method for hypothesis testing and the last part says that "if the p-value is not small enough to rule out chance, we fail to reject null hypothesis". A p-value is stated as: the probability of observing a value of the test statistic at least as contradictory to the null hypothesis as the one computed from the sample data"

Now, what doesn't make sense to me is that if p-value tells me the probability of observing a value of a test statistic at least as contradictory to the null hypothesis, then surely a low p-value(less than significance) would mean that it isn't very contradictory and thus we fail to reject but in fact they do it the other way around. I.e if p-value is less than significance, they actually reject null hypothesis?

Some sites say that the p-value is the strength of the evidence from the test statistic. Any clarificaiton?

Answer 1

Let's use a simple example. You obtain a group mean of 10. Your H0 hypothesized or reference mean is 0. Is 10 sufficiently different from 0 to declare statistical significance? Well that depends on the variability (and possibly other factors like N) inherent in your data, and that then dictates your p-value.
Smaller p-values do suggest more 'evidence' against the null hypothesis, but be careful - an effect either is or is not significant under the NHST regime (c.f. The Neaman-Pearson approach of effect sizes).

And here's where your confusion lies: the p-value is the probability of observing such an extreme (or more) mean by chance alone. So lower values mean that it's less likely to have happened randomly.

Answer 2

Many people attempt to simplify the definition of a p-value, and inevitably subtle errors or misunderstandings seep into their discussions. It's best to resist the urge, and speak precisely as possible:

A p-value is the conditional probability of observing a statistic equal or more extreme than the value produced by the data actually collected, given all the assumptions of the null hypothesis.

That's it.

The general procedure for producing such a thing is:

• State precisely a null hypothesis.
• Use the null hypothesis to create a probabilitic model of how your data would be generated if the null hypothesis is true.
• Create a measure of the "extremeness" of your data, this is the test statistic.
• Use the probabilistic model from the second bullet point to derive a distribution of the test statistic from the third.
• Compute the conditional probability from the definition of p-value.

The standard tests that people learn in school are just patterns of the above procedure that tend to come up repeatedly in various situations.