2
$\begingroup$

I have a large dataset (the population) and a large subset of it (the sample) containing the same, continuous variables. The sample represents more than 90% of the population but is not random -- we would like the sample to be 100%, the difference is due to systematic problems.

I would like to measure the extent to which the distribution of each variable in the sample has shifted compared with its distribution in the population. This would help inform the analysts using our sample about whether they need to correct for any distortions our process might have introduced (the analysts are not allowed to access the population).

I've looked at using a two-tailed t-test and/or a Kolmogorov-Smirnov test (which was suggested in an answer here) but in both cases the setup is that you have two samples from an unknown population with a theoretical distribution. In my case I have the actual population, so I feel like there might be a better method.

I should add that these datasets have 100s of millions of rows with 1000s of variables, so computationally intensive options are not feasible. So the real problem is to identify one or more "canary in the coalmine" statistics that are cheap to calculate and will flag up a small subset of problematic variables for further analysis.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Why can't you use a Kolmogorov-Smirnov test? The "theoretical distribution" is the empirical distribution of the full population and your test distribution is the distribution of the sample. As you do multiple tests, you will need a multiple testing correction. $\endgroup$
    – cdalitz
    Commented Jun 30, 2023 at 8:20
  • $\begingroup$ @cdalitz That was my thought too but I can't really see how to get an empirical distribution from the population without something computationally heavy like a KDE. The variables are likely to have a wide variety of distributions so fitting Gaussians (or similar) onto them doesn't seem promising. $\endgroup$
    – helveticat
    Commented Jun 30, 2023 at 8:33
  • $\begingroup$ @cdalitz Actually, sorry, it's this isn't it? en.wikipedia.org/wiki/Empirical_distribution_function. I hadn't seen this technique before but it seems obvious now. This might be a good solution for us. $\endgroup$
    – helveticat
    Commented Jun 30, 2023 at 8:36
  • 1
    $\begingroup$ Using the empirical distribution function actually is the point of the KS test: In the one-sample version, the empirical CDF is compared to the theoretical CDF, whereas in the two-sample test two empirical CDFs are compared. The KS test only uses the maximum difference between the CDFs, whereas the Cramér-van-Mieses test uses the (squared) area between the CDFs and its test statistic is thus more sensitive to differences, but also computationally more complex to compute. $\endgroup$
    – cdalitz
    Commented Jun 30, 2023 at 9:16
  • 1
    $\begingroup$ Using the methods I describe at stats.stackexchange.com/questions/35220, in a single pass over the data you can hugely reduce the storage needed to represent the relationships between the sample and the population for all variables with high precision (as QQ plots). It would then be efficient to post-process those representations using any complex, sophisticated, or computing-intensive method you like. To the extent the sample is like the population, this approach will be extremely efficient: each representation might be as small as a handful of ordered pairs. $\endgroup$
    – whuber ♦
    Commented Jun 30, 2023 at 15:21

1 Answer 1

Reset to default
1
$\begingroup$

Yes, for testing whether the distribution in your sample is suspicously different from the known distribution, you can use the Kolmogorov-Smirnov test. It is based on the distance between the empirical dsitribution functions (ECDF) of the theoretical distribution (the known distribution) and the test distribution (your sample).

As there are different distance measures for distribution functions, different tests are possible: The Kolmogorov-Smirnov test uses the maximum difference between the ECDFs (aka "supremum distance") as a test statistic, whereas the Cramér-van-Mieses test uses the squared area between the ECDFs (aka "$L_2$ distance"). As you test for many variables, you should use a correction for multiple testing, because with an increasing number of tests the probability of finding at least one "significant difference" quickly approaches one.

A different, visual method to compare the ECDFs is by means of a QQ-plot, as @whuber suggested. The closer it is to a straight line, the more similar the distributions are.

Beware, however, that these are only tests whether the isolated (or "marginal") distributions of the variables are similar, not whether your sample represents (multi-) correlations in the original distribution correctly. One method to check for this could be to compute confidence intervals for all entries in the correlation matrix of your sample data and check how many true correlations (computed from the known distribution) fall outside these confidence intervals. For a confidence level of $1-\alpha$, this should be not much more than a fraction $\alpha$.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot, this is very helpful context and makes me feel more confident about the K-S approach. $\endgroup$
    – helveticat
    Commented Jul 3, 2023 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.