The p-value is defined as the probability, under the assumption of hypothesis H, of obtaining a result equal to or more extreme than what was actually observed (Wikipedia). By "a result" it is intended a particular statistic.

For example, if I want to fit a distribution of data ($N$ items), I would choose the parameters that maximize the likelihood. Then, if I want to compute the p-value I can use the Kolmogorov-Smirnov distance: I compute it between the CDF of the empirical distribution and the CDF of the fit; then I sample datasets of $N$ items from the fit distribution, fit them, and compute the KS between the samples CDF and the CDF of the fits of the samples. Finally I estimate the probability that the KS of data-fit is bigger than the KS from samples, which would be the p-value.

Now, the KS is a distance between two distributions. Can I use the Kullback-Leibler divergence instead of the KS? In this case, it would be the Likelihood itself. So I would sample from the fit distribution, apply maximum likelihood and estimate the probability of the likelihood of the data w/r the model to be as extreme as the likelihood of sampled datasets w/r to their model.

What would be the problem with this approach? Is the $\chi^2$ analysis the same for gaussian-distributed data? Can I do the same for other distributions on the basis of the Wilks' Theorem?

If is not clear, please tell me, I can expand/explain.

  • 1
    $\begingroup$ You lost me at computing the p-value: that's not how it is done when you fit data using Maximum Likelihood. You have abruptly changed your procedure, suddenly abandoning one approach (with its set of assumptions) and replacing it with another (with a completely different set of assumptions) that is, BTW, invalid. Given that you are mixing and matching such different and incompatible approaches, I can't imagine what you are referring to by a "$\chi^2$ analysis" or the application of Wilks' Theorem. $\endgroup$ – whuber Aug 10 '15 at 14:28
  • $\begingroup$ to compute the p-value, shouldn't I sample the statistic that I measure, given that the model is true? (in this case the KS) I think the approach has not changed, can you explain me which would be the two approaches? $\endgroup$ – chuse Aug 10 '15 at 14:33
  • $\begingroup$ ML does not have either a null hypothesis or a statistic. It can be used for hypothesis testing when comparing ML fits of nested models. In any event, ML assumes a finitely parameterized distribution (it is "parametric"), whereas KS makes no such restriction (it is "nonparametric"). It sounds a little like you would like to run a goodness-of-fit test for a ML estimate, but your references to p-values, $\chi^2$ analysis, and so on make it hard to be sure. $\endgroup$ – whuber Aug 10 '15 at 15:56
  • 1
    $\begingroup$ Yes, so I would propose a parametric distribution, and estimate its parameters through ML. Then, I make an hypothesis test, where the null hypothesis is that the data is sampled from the distribution with the estimated parameters. In order to do that I compute the p-value of this hypothesis. You think this is not a valid approach? $\endgroup$ – chuse Aug 11 '15 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.