2
$\begingroup$

I've been using the two sample Kolmogorov-Smirnov test to compare observations from 2018 vs observations from 2019. I heard from my co-worker that you can interpret the D-statistic as the magnitude of the observed differences and the p-value as the likelihood of the null hypothesis having observed such magnitude of differences or greater.

Is that interpretation correct? It certainly is easier to explain to management.

Edit 1: Per advice from Glen_b, Added two new points to interpretation of p-value

Edit 2: Semantics change about null hypothesis

Improve this question
$\endgroup$
1
  • $\begingroup$ In relation to the initial question, this is somewhat related $\endgroup$
    – Glen_b
    Commented Jul 25, 2019 at 23:28

1 Answer 1

Reset to default
2
$\begingroup$

The interpretation is not correct.

The $D$ statistic is the largest observed difference in ecdf, and as such is a way to describe the magnitude of the difference in distribution (one of many possible ways). Which is to say, your colleague's description of $D$ is more or less correct.

However a p-value is not the likelihood (nor even the probability) of the observed measure of difference.

(In the present context) It's the probability of observing a $D$ statistic at least as large as yours, when the two samples were drawn from the same continuous population.

Those two things (what you report your colleague as saying and what I wrote) are quite distinct notions.

May find it useful to read this (and the actual ASA Statement on Statistical Significance and P-Values itself, which immediately follows; each is only a couple of pages long):

Ronald L. Wasserstein & Nicole A. Lazar (2016),
The ASA Statement on p-Values: Context, Process, and Purpose,
The American Statistician, 70:2, 129-133,
DOI: 10.1080/00031305.2016.1154108
http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108#.Vt2XIOaE2MN

It does explain what a p-value is and what it isn't, though it doesn't address all the ways in which people misunderstand p-values. It also doesn't list all the criticisms of p-values nor everything people may do to supplement or replace them. (This is not intended as a criticism, just a description of some of what it contains.)

You might also get some value from

Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Lazar (2019),
Moving to a World Beyond “p < 0.05”,
The American Statistician, 73:sup1, 1-19,
DOI: 10.1080/00031305.2019.1583913
https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913

if somewhat less directly relevant to the present question. (I don't necessarily agree with everything in this article but that's not necessarily a problem.)

"Likelihood" in ordinary conversation is more or less interchangeable with probability, but in statistics they're quite distinct concepts.

There were three issues with what you said in relation to the p-value; one was using the word likelihood when you mean probability; the second was the probability you gave related to the wrong event (which you've fixed in your restatement); and the third was it's actually the probability of that event given that the null hypothesis is true (i.e. given that the populations that the samples are drawn from are the same) -- and given the usual assumptions of the test, naturally.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks. What if I revised my last sentence as "the p-value as the likelihood of the observed differences or greater?" $\endgroup$
    – Jack
    Commented Jul 23, 2019 at 15:54
  • $\begingroup$ See the edit .. $\endgroup$
    – Glen_b
    Commented Jul 23, 2019 at 22:40
  • $\begingroup$ Hi Glen. Thanks for clarifying. I tried to revise my answer to address your first and third points. Can you tell me if I got it right this time? $\endgroup$
    – Jack
    Commented Jul 25, 2019 at 15:52
  • $\begingroup$ 1. No; it's not in any sense the probability of the null hypothesis. I also advise you to avoid the word 'likelihood' when you mean probability. People try every day to "simplify" the definition of a p-value ... and mislead every time they do so. The informal definition given at the linked page is correct: "Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (e.g., the sample mean difference between two compared groups) would be equal to or more extreme than its observed value." $\endgroup$
    – Glen_b
    Commented Jul 25, 2019 at 17:41
  • 1
    $\begingroup$ 2. Please note that if you change your question after I answer it, you render things I wrote in my answer nonsensical; my answer responds to a question that no longer exists. A question is not meant to be a moving target; you posted a question and I answered that question. The task of an answerer is not to keep answering a series of new versions of your questions; that's a conversation. $\endgroup$
    – Glen_b
    Commented Jul 25, 2019 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.