Finding the best-fitting parametric distribution for empirical data

I have an empirical dataset of samples that I'm trying to fit a (continuous) parametric distribution to from a list of possible distributions. I'm not trying to fit the parameters of a specific distribution.

Is there a "smart" (statistically speaking) method to do this? I've thought of the following:

1. For each distribution, find the best parameters using for example MSE.
2. For each distribution + best fitted parameters use Kolmogorov–Smirnov test or Pearson's chi-squared test and choose the one with the highest score.
3. For each distribution + best fitted parameters use Pearson's chi-squared test and choose the one with the highest score.

Which of the above would be the best approach? In 1, why not use the MSE as a criterion? Why do we need these tests?

Thanks.

• You should check the Wikipedia entry on overfitting to understand why a direct comparison of MSE's is not a good idea... Dec 26 '14 at 20:31
• On statisticshowto.com/kolmogorov-smirnov-test, the limitations of the KS-test are: "In order for the test to work, you must specify the location, scale, and shape parameters. If these parameters are estimated from the data, it invalidates the test. If you don’t know these parameters, you may want to run a less formal test (like the one outlined in the empirical distribution function article).". Also, it says: "it is commonly used as a test for normality to see if your data is normally distributed." For this reason, some might be more interested in a RMSE method based on the histogram. Nov 3 '20 at 7:25
• Other empirical distribution function (EDF) goodness-of-fit tests, that really are tests for normality, are outlines next to the KS-test here: support.sas.com/documentation/cdl/en/procstat/63104/HTML/… Nov 3 '20 at 9:06

I have a set of samples that I'm trying to fit a (continuous) distribution to from a list of possible distributions.

The main question you need to deal with is why do that?

What are you ultimately trying to achieve? What does this do for you that you can't get some other way? Why would your data come from some such list?

Is there a "smart" (statistically speaking) method to do this?

Yes: In general, don't. It's going to result in overfitting.

However, if you must do this, you should go to some length to avoid assessing the suitability of the distributional model on the same set (or subset) of data you use to select between the models. (It's also very important you not use that same data for inference involving parameters.)

Sample splitting/cross validation will get you a long way.

For each distribution, find the best parameters using for example MSE.

Why would MSE be a good choice?

For each distribution + best fitted parameters use Kolmogorov–Smirnov test

1) K-S assumes a specified, not fitted distribution.

2) you're presumably comparing distributions with different numbers of parameters; why should they be comparable (models with more parameters should give a better fit -- if this is your criterion you'll simply choose a model with lots of parameters)

3) a test answers the wrong question (the question is whether the model is reasonable for some as-yet-unstated purpose, not whether some hypothesis test rejects or fails to reject)

Does that make any sense?

I think it's ill-advised, but there's not presently enough information to provide a lot of guidance toward something better.

In 2, why not use the MSE as criterion?

First, what are you calculating MSE of? Why is the MSE a good measure of your loss-function?

Why do we need these tests for?

It's not clear what you're asking here. I don't think they serve any useful purpose for your kind of problem. Why would you use them for this?

The problem with K-S test is that it won't work out of the box in cases where you are estimating the theoretical distribution parameters from the sample, see item #3 here. There is a way to overcome this issue through Monte Carlo simulations, see here how to do it.

You can use $\chi^2$ goodness-of-fit test. It has its own issues, such as dependence on bins.

I don't think any of these tests would be good to automatically select the best fit. I would rather run the tests to reject the distributions that don't fit well, then deal with those which are left to pick one of them manually based on the knowledge of the underlying phenomenon.

• The comparison of models by likelihood ratio tests is a well-established and well-documented approach. Dec 26 '14 at 20:33
• @Xi'an, are you going to apply LR test to pick between different distributions? It's designed to test nested models. Dec 26 '14 at 20:36
• by defining an encompassing model, it allows to consider both models as submodels of the encompassing model. Dec 26 '14 at 21:02
• @Xi'an, let's say he wants to test 30 different distributions, how would you do it with encompassing? Dec 26 '14 at 21:07