2
$\begingroup$

I have a large sample size of about 50,000

I want to determine what theoretical distribution fits the distribution of my samples the best. What I did is to fit all distributions I know to the data and check which one has the lowest SSE. I'm personally quite happy with this as the distributions with low SSE also seem to fit the data graphically quite well.

Anyhow, I've realized that most researchers use a statistical test to "proof" that one distribution fits better than another. I thought maybe I could use the KS-Test on the best distributions and compare the p-values. On the other hand the sample size is quite large and I've heart in cases like this effect size might be more interesting. So I thought of comparing the different distributions using the Kullback–Leibler divergence.

What do you think?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Your concern about using statistical tests for fitting to distributions are spot on because they can be overly sensitive for large sample sizes. This is because the tests are trying to look for any resolvable discrepancy and more samples gives them more power to do that with. See this answer here for a nice explanation: Is the Kolmogorov-Smirnov-Test too strict if the sample size is large?

So in your case if you are satisfied with the graphical fit of the data to the distribution you should be good to go. I have a few other suggestions you can try and that is use the best fit distribution to sample 50,000 data points (or however many you have) and then compare your sample to your data to see if things like the mean, median, standard deviation, etc match (or are at least close enough for the work you're doing). If those all hold then I would say you are good to go.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks a lot for your fast reply. I think that is a good advise but I would like to compare the goodness-of-fit using one specific value. Is there anything I could use in this case? What do you think about using the SSE? $\endgroup$
    – MajinBoo
    Commented Oct 14, 2019 at 13:46
  • $\begingroup$ I think the SSE is a fine metric to use in this case. There are some problems with it in that some distributions are more complex (have more parameters) than others. But besides that it should be fine. That said: Something else to consider doing is thinking about what kinds of distributions make sense for your data. Are there restrictions on your data that the probability distribution needs to obey as well (e.g., data is always positive?). $\endgroup$
    – Patrick
    Commented Oct 14, 2019 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.