Estimating Probability Density for Sample

Question

I have a dataset of over 20,000+ samples. The objective here is to define a distribution for the sample so that I can plot all possible outcomes. However, I am unable to find an appropriate distribution I can use to estimate the Probability Density. I have tried to test the sample with Normal, Cauchy, Laplace, Student's T and Weibull. In all cases, Kolmogorov-Smirnov Test rejects possibility of my sample following any of the mentioned distributions. I have also tried to estimate using KDE, with not a very promising result. I tried using the KS test to check similarity between the KDE and my sample and even here similarity is rejected. I am stuck on what should be my next step to estimating the Probability Density.

Answer 1

Equality-null hypothesis tests are entirely the wrong tool for model selection.

People only tend to notice when sample sizes are very large. However, they usually draw the wrong conclusion in that instance; rather than realizing that the test they used doesn't answer the question they needed an answer to, they decide that hypothesis tests are broken.

Pushing up p-values by sub-sampling the data is a commonly used strategy in response to the problem, but it's quite misguided — arbitrarily throwing out information doesn't solve the problem, it merely disguises the issue, leaving the underlying problem (that it's the wrong tool for the job in any case) untouched.

The reason why the p-values are small is not just that the sample size is large — it's that no simple form distributional model is correct. Correctness is not even the purpose of models.

In short, to quote George Box "Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful". Models should be chosen - when they must be chosen at all - with regard to their purpose.

You say you need "to define a distribution for the sample so that I can plot all possible outcomes" but the approach you have used doesn't solve that problem. You're choosing between models that define the distribution outside the range of the data, sure, but there's no guarantee whatever that the process follows any of your arbitrarily-selected laundry list of distributions where you don't have data, and no obvious reason to imagine they should.

If the empirical distribution of the data themselves will not serve because you want a distribution where you don't have data, then your problem is that you demand information where you have none. Selection from the arbitrary laundry list of distributions you happen to choose to select between does not solve that problem either — again, it just disguises it. In short, that, too, is using the wrong tool for the task.

If you really want to know about the distribution where there are no data you have to bring in some external information, whether from theory, consideration of the processes involved, other data, etc. It requires more effort than presuming that fit over here implies fit way over there, but it has a somewhat better hope of avoiding potentially very misleading answers.

Answer 2

You are choosing distributions that match the empirical distribution. This will result in smooth estimates but the precision will be inherited from the variance of the empirical estimates. So there is little to be gained over settling with the empirical cumulative distribution function, spike histograms, and kernel density estimates.

Answer 3

When the number of samples is high, the null hypothesis is rejected if any distribution other than the true distribution, even if slightly different, is included in the null hypothesis. It is because the differences become more obvious as the sample size increases and type-II error decreases. If you repeat the test based on a small number of data from the same data set, the null hypothesis will not be rejected. In this situation, when there is enough data, you should not worry about very small p-values, because any other distribution will be rejected again unless you can guess the true distribution or it has been given (where you can often accuratly estimate its parameters as you have a large sample). In these cases, the selection of the distribution can be done based on the best fit, if you have a short list, for example, you can pick the distribution that has the minimum test statistic value or the least Leibler-Kullback divergence from the empirical distribution.

Answer 4

You have a sample with 20,000+ data points. That is a "huge" sample. Any test against a theoretical distribution, with such a large sample, will fail, no matter what the truth is. Which is why all your p-values are 0.000 (did you not find that odd?). See e.g. here for normality tests on CV, but this would apply to any similar Q-Q test (because real world data is never one of these perfect mathematical distributions).
Now, a possible sugestion? Take a much smaller sub-sample from your 20,000+ data points, say only 100-200 (so maybe every 100 or so observation?), or even take several such small sub-samples, and try again various theoretical distributions. I feel rather confident that several will give you large-ish p-values; then pick the one which had p-values above some threshold for all/most of the sub-samples.
Last, a comment; your sample is very "well-behaved"; symmetrical, uni-modal, centered on 0, and (very) large. For any analysis you may want to do beyond your question, you might just as well assume it is normal (which you did not test? but it would have failed as well).