5
$\begingroup$

I have fitted two models, one with a series of parameters $\theta_i = 0$, and one with those parameters $\theta_i \in \mathbb{R}$, with Cauchy priors.

I want to compare whether the addition of the extra parameters improves the model in a statistically significant way, so I perform a likelihood ratio test.

Can I use the training data to do this?

I have done leave one out cross validation instead of a regular test and training set, so if I performed the LRT on the test data, it would be comparing sets of $n$ different models, so things might get odd?

Thanks!

Improve this question
$\endgroup$
1
  • $\begingroup$ @Ben ah this clears things up, thanks! So I should run the LRT on the training data, and then if there's no evidence of overfitting when I look at the test data, it'll hold for that too? $\endgroup$
    – A F
    Commented Jun 6, 2022 at 12:02

1 Answer 1

Reset to default
3
$\begingroup$

Likelihood ratio tests are done all the time (in the non-precise meaning of the phrase) on training data, e.g., F-tests for whether a regression is significant or t-tests for differences between means. So yes, you can certainly use all of your data for a likelihood ratio test.

However, one assumption of the LR test is that the fitted model(s) are fit using maximum likelihood. In your case, the larger model is fitted using Bayesian techniques, not ML, while your smaller model is just the constrained version of an MLE (with all the estimates being set equal to zero), which is fine. You clearly have a likelihood function in hand, otherwise your Bayesian approach wouldn't have been possible, so, in order for the asymptotic distribution of the likelihood ratio (or associated statistic) to hold, you should fit the larger model using maximum likelihood.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! Both my models actually have Cauchy priors on all the parameters. I get a bit confused with terminology sometimes - I've got posterior distributions, and so I've found parameters which maximise the posterior likelihoods (posterior modes). Is this not ok? $\endgroup$
    – A F
    Commented Jun 6, 2022 at 12:01
  • $\begingroup$ It's fine from an estimation point of view, but likelihood ratio techniques don't use priors, just the likelihood functions, and their distributions, asymptotic or otherwise, are derived under the assumption that you are maximizing the likelihood. Perhaps a Bayesian counterpart would be better for your purposes? vasishth.github.io/bayescogsci/book/… might help... $\endgroup$
    – jbowman
    Commented Jun 7, 2022 at 15:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.