Should I sub-sample very large datasets to run the Kolmogorov-Smirnov (KS) test? I have two lists with many millions of values (one has 21,410,024,757 values and the other 10,561,427 values). When I run the two-sample KS test the p-value is 0, but many posts suggest that for such large datasets the KS test isn't reliable, e.g:
Kolmogorov-Smirnov test statistic interpretation with large samples
Is it a good approach to sub-sample the two lists of values randomly and uniformly?
 A: In effect what you are asking for is a p-hack.
The KS test is not "unreliable" with large $n$. Quite the opposite. It is overly powered. What does the KS tell you? If two distributions are different. Lots of data means lots of power. And you now have evidence that the first population with a sample of 21 million is different than the second population with a sample of 10 million. I don't think we should be surprised by this.
The problem is that you didn't think about this before running the test. If you now set the alpha level to some very low value, you are forced to decide whether to set this value to call the test significant or non-significant, you already know the p-value. It begs the question why run the test at all?
Also a note about reporting p-values. A p-value is never 0. Even when the sample is identical in the two arms, as a formality, you should report the p-value as p < 0.001 (or whatever significant figure you choose to report p-values); declaring something to be impossible can lead to a serious diagnosis of foot-in-mouth syndrome.
I find that tests are not useful summaries of distributional differences. It's surprising how informative graphics can be to summarize these values instead. Consider simply showing two overlayed estimates of the density using a filtering process. That was if the "difference" leading to a significant result is some small, ignorable facet, then the graph tells us there is no practical difference to mind.
A: Can you say the source of the data and what practical goal you hope to reach doing a K-S test on these two huge samples.
in particular:

*

*With such large datasets, you should be able to see some obvious differences between the the two samples, unless they are essentially from the same distribution,


*Why would a P-value of $\approx 0$ unsatisfactory for your purposes?


*A major complaint about the K-S test is that it sometimes detects differences not of practical importance when sample sizes are very large.
Consider the following, using R:
set.seed(2021)
x1 = rgamma(10^6, 3, .1)
x2 = rgamma(10^5, 3, .1)
x3 = rgamma(10^5, 3, .101)
ks.test(x1, x2)

        Two-sample Kolmogorov-Smirnov test

data:  x1 and x2
D = 0.003331, p-value = 0.2654
alternative hypothesis: two-sided

ks.test(x1, x3)

        Two-sample Kolmogorov-Smirnov test

data:  x1 and x3
D = 0.008329, p-value = 6.656e-06
alternative hypothesis: two-sided

mean(x1); mean(x2); mean(x3)
[1] 30.00744
[1] 30.0714
[1] 29.70087
boxplot(x1,x2,x3, col="skyblue2", horizontal=T)


A: The problem with your proposed solution is that there is no good way to determine how much downsampling you should do to get the "right" answer.  Reducing the sample size will reduce the power of the test, but you have no idea how much reduction is needed.  The deeper problem is that the K-S test is telling you whether there is statistical evidence for a difference between the distributions, but what you want to know is, whether the difference is important enough to be concerned about it.  The K-S test can't tell you this.
The first thing you are going to have to do to answer that question is to decide what kinds of differences you care about.  Do you want to know if the means are different?  The variances?  The 95th percentiles?  Statistics can't answer this for you.  You have to do some thinking about what kinds of differences would cause you to do something different in your business.  Then devise tests for those differences specifically.
What form the tests you are looking for are going to take depends a little on what kinds of differences you are looking for, but generally speaking you are going to want to avoid hypothesis tests and look for estimates of confidence intervals for the population parameters you are interested in.  So, for example, if you are interested in whether the mean of one distribution exceeds the other by some particular margin (which you have predetermined is relevant to your business use), you might start with the standard error of the means of the distributions and use them to estimate the standard error of the difference between the means.  If you're interested in something more esoteric, you might have to fit some parametric distributions to the data, or use a bootstrap Monte Carlo.  Whatever you method you pick, you will need to keep its assumptions and limitations in mind and make reasonably certain that your data doesn't violate them.
The most important thing, however, is to start by thinking about what it is you are really looking for.  Why do you care whether the distributions are different?  Trying to run statistical tests before you've thought about that is futile.
A: I wonder, how you have such a huge data. Any way, as I understand, your primary interest is to know whether the distribution pattern is same in both the data or not?
I think, it is better, if you resort to some sort of sampling.
it is not always possible to enumerate the entire population or huge population and as is not cost-effective, sampling is always advocated.  I suggest that you have relatively a small random sample of the data ( say 10,000 each) and then go for the analysis.
Form a frequency distribution for both the sampled data and then go for K-S test or the Chi-square test, as desired.
