0
$\begingroup$

I am doing a log-likelihood ratio test between seven models fitting a set of data with N=2 000 000. The models are nested; each one contains the same parameters as the last, and some more parameters in addition. You can think of it like simply doing higher-order polynomial fits. Each higher order model has 3 more parameters than the last.

I'm trying to use the log-likelihood ratio test to prevent overfitting. However, the likelihoods I'm getting all have differences on the order of 100, or even 1000, even though each model only differs from the last by 3 degrees of freedom. As such, with such large LRT statistics, I'm getting miniscule p-values each time I move from a simpler model to a more complex one.

If I randomly take 300 samples from my set, the LRT statistic is more on the order of 1 or 10. But every time I take a different 300 samples, I get different results about which model "passes" the test and which does not.

It's the same if I take different subsets of n=1 000 000 from my original set; the ordering of the log-likelihood values changes every time.

Is the log-likelihood really that unstable when it comes to sample size? Am I doing something wrong? Is there a way to normalize log-likelihood? (I'm finding for a system that has a continuous distribution of possible datasets, the answer is no)

I hope this is clear. I will try to answer any questions as best I can.

|cite|improve this question
$\endgroup$
0
$\begingroup$

The log-likelihood is not unstable, but with such a huge sample sizes, any tiny departure from your null value will get picked up by the test statistic. A classical example where you should also consider the effect size rather than the p-value alone.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Two questions: if the log-likelihood (not the test statistic, just the literal max log-likelihood) is not unstable, why is it changing so dramatically when the sample size changes? Second, how would I use effect size in this case (i.e. when comparing models)? $\endgroup$ – Meredith Aug 30 '18 at 16:56
  • $\begingroup$ 1) The log-likelihood is the sum of the logged probabilities of your outcome over all observations. The more observations you have, the smaller it becomes. 2) You could look at the magnitude of the parameters you add, and see if they are large enough to be relevant in practice. $\endgroup$ – Knarpie Aug 31 '18 at 7:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.