I am doing a log-likelihood ratio test between seven models fitting a set of data with N=2 000 000. The models are nested; each one contains the same parameters as the last, and some more parameters in addition. You can think of it like simply doing higher-order polynomial fits. Each higher order model has 3 more parameters than the last.
I'm trying to use the log-likelihood ratio test to prevent overfitting. However, the likelihoods I'm getting all have differences on the order of 100, or even 1000, even though each model only differs from the last by 3 degrees of freedom. As such, with such large LRT statistics, I'm getting miniscule p-values each time I move from a simpler model to a more complex one.
If I randomly take 300 samples from my set, the LRT statistic is more on the order of 1 or 10. But every time I take a different 300 samples, I get different results about which model "passes" the test and which does not.
It's the same if I take different subsets of n=1 000 000 from my original set; the ordering of the log-likelihood values changes every time.
Is the log-likelihood really that unstable when it comes to sample size? Am I doing something wrong? Is there a way to normalize log-likelihood? (I'm finding for a system that has a continuous distribution of possible datasets, the answer is no)
I hope this is clear. I will try to answer any questions as best I can.