1
$\begingroup$

I'm an applied math graduate student with little background in statistics. I'm currently dealing with the following problem: I have a set of samples that I'm trying to fit a (continuous) distribution to from a list of possible distributions. I'm not trying to fit the parameters of a specific distribution.

Is there a "smart" (statistically speaking) method to do this? I've thought of the following:

  1. For each distribution, find the best parameters using for example MSE.
  2. For each distribution + best fitted parameters use Kolmogorov–Smirnov test or Pearson's chi-squared test and choose the one with the highest score.

Does that make any sense? In 2, why not use the MSE as criterion? Why do we need these tests for?

Thanks.

$\endgroup$
  • 2
    $\begingroup$ You should check the Wikipedia entry on overfitting to understand why a direct comparison of MSE's is not a good idea... $\endgroup$ – Xi'an Dec 26 '14 at 20:31
2
$\begingroup$

I have a set of samples that I'm trying to fit a (continuous) distribution to from a list of possible distributions.

The main question you need to deal with is why do that?

What are you ultimately trying to achieve? What does this do for you that you can't get some other way? Why would your data come from some such list?

Is there a "smart" (statistically speaking) method to do this?

Yes: In general, don't. It's going to result in overfitting.

However, if you must do this, you should go to some length to avoid assessing the suitability of the distributional model on the same set (or subset) of data you use to select between the models. (It's also very important you not use that same data for inference involving parameters.)

Sample splitting/cross validation will get you a long way.

For each distribution, find the best parameters using for example MSE.

Why would MSE be a good choice?

For each distribution + best fitted parameters use Kolmogorov–Smirnov test

1) K-S assumes a specified, not fitted distribution.

2) you're presumably comparing distributions with different numbers of parameters; why should they be comparable (models with more parameters should give a better fit -- if this is your criterion you'll simply choose a model with lots of parameters)

3) a test answers the wrong question (the question is whether the model is reasonable for some as-yet-unstated purpose, not whether some hypothesis test rejects or fails to reject)

Does that make any sense?

I think it's ill-advised, but there's not presently enough information to provide a lot of guidance toward something better.

In 2, why not use the MSE as criterion?

First, what are you calculating MSE of? Why is the MSE a good measure of your loss-function?

Why do we need these tests for?

It's not clear what you're asking here. I don't think they serve any useful purpose for your kind of problem. Why would you use them for this?

$\endgroup$
0
$\begingroup$

The problem with K-S test is that it won't work out of the box in cases where you are estimating the theoretical distribution parameters from the sample, see item #3 here. There is a way to overcome this issue through Monte Carlo simulations, see here how to do it.

You can use $\chi^2$ goodness-of-fit test. It has its own issues, such as dependence on bins.

I don't think any of these tests would be good to automatically select the best fit. I would rather run the tests to reject the distributions that don't fit well, then deal with those which are left to pick one of them manually based on the knowledge of the underlying phenomenon.

$\endgroup$
  • 1
    $\begingroup$ The comparison of models by likelihood ratio tests is a well-established and well-documented approach. $\endgroup$ – Xi'an Dec 26 '14 at 20:33
  • $\begingroup$ @Xi'an, are you going to apply LR test to pick between different distributions? It's designed to test nested models. $\endgroup$ – Aksakal Dec 26 '14 at 20:36
  • 1
    $\begingroup$ by defining an encompassing model, it allows to consider both models as submodels of the encompassing model. $\endgroup$ – Xi'an Dec 26 '14 at 21:02
  • 1
    $\begingroup$ @Xi'an, let's say he wants to test 30 different distributions, how would you do it with encompassing? $\endgroup$ – Aksakal Dec 26 '14 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.