2
$\begingroup$

Example

The example is made up because I hope that it’s more accessible than my actual problem.

I want to determine the number of planets of a star. I have:

  • data for some astronomical observable of that star, e.g., an intensity time series,
  • a model that describes the observable with the sizes of the star’s planets as parameters,
  • goodness of fits / Bayes factors for the planet sizes (parameter values) that best explain the data for a given number of planets,
  • prior data on how likely a given number of planets is,
  • no prior data on planet sizes.

The model is such that for an infinite number of planets, it can explain any data. Also, a planet with size zero is equivalent to no planet.

Question

If I have priors on the number of parameters itself, how do I properly penalise additional parameters to avoid overfitting? In the example, how do I penalise an extra planet? I am primarily skeptical as to whether my prior information on the number of planets already covers any extra penalty imposed by an information criterion such as the Akaike or Bayesian information criterion.

Intuitively I would say that it isn’t covered and thus I would use an information criterion with the product of Bayes factor and the known priors for the number of planets as likelihood. My prime rationale for this is that in absence of prior information on the number of planets, i.e., equal priors, I clearly need to have an additional penalty lest I arrive at a model with an infinite number of planets.

On the other hand, I was thinking: What if the ratio between the prior probabilities of one and no planet was higher than the parameter penalty imposed by the information criterion? For example, in case of the AIC, what if one planet were more probable than no planet by a factor $e$ (from prior insights)? In that case, my approach would always favour one planet over none – unless I implement a penalty, lower prior, or similar for very small planets.

Improve this question
$\endgroup$
3
  • $\begingroup$ I think the vagueness of the example makes it harder to discern the question. I would have thought that more planets would mean more parameters to describe them. As you already have a prior on the number of parameters, if it penalises large models, it is already helping to avoid overfitting, Note also you may want to avoid overfitting by penalising the values of parameters (e.g. regularisation). The idea that complexity is only measured by the number of parameters is a bit past its sell by date. $\endgroup$
    – Dikran Marsupial
    Commented Sep 13, 2022 at 13:21
  • $\begingroup$ @DikranMarsupial: I would have thought that more planets would mean more parameters to describe them. – They do. Each planet comes with one extra parameter, namely its size. (I hoped this was clear from the question.) — As you already have a prior on the number of parameters, if it penalises large models, it is already helping to avoid overfitting – Well, if you so wish, my question is whether this is sufficient or some information criterion needs to be included on top. The example when my prior says that one planet is more likely than none suggests this. $\endgroup$
    – Wrzlprmft
    Commented Sep 13, 2022 at 13:33
  • $\begingroup$ there is no way of telling whether a prior on the number of parameters is sufficient to avoid overfitting without trying other forms of overfitting avoidance and seeing what happens. The more (correct) prior knowledge you put into a model, the more likely it is to perform well, but if you have enough data your priors become irrelevant - so there is a compromise that also involves data quality and quantity. If you want to avoid ambiguity, then set out the model formally with equations. $\endgroup$
    – Dikran Marsupial
    Commented Sep 13, 2022 at 13:51

1 Answer 1

Reset to default
2
+50
$\begingroup$

You are mixing two different concepts: model selection and regularization.

  • AIC and BIC are used for model selection. Model selection is not that different between frequentist and Bayesian settings, though they may use different tools and methods. You can easily search for Bayesian model selection to learn more.
  • Another thing is regularization. In a frequentist setting, you could use regularization to penalize the number of parameters (e.g. ridge or lasso regression), the same is done in a Bayesian setup with choosing sparsity introducing priors that drag the parameters towards zero. You either need to choose the priors based on what you know about the problem or choose a hyperprior that would be used to learn the priors.
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your answer. I haven’t delved into the links and concepts you refer to, but I definitely will later. Still, I expect that it would be useful if you could briefly elaborate what approach is appropriate for the problem in question, where the choice of model somewhat aligns with the scientific question (i.e., how many planets there are). $\endgroup$
    – Wrzlprmft
    Commented Aug 27, 2022 at 9:53
  • $\begingroup$ @Wrzlprmft there’s no one appropriate solutions, but many possibilities. What you are describing sounds like a research project that can take several weeks or months, than something that has one, simple solution. $\endgroup$
    – Tim
    Commented Aug 27, 2022 at 10:04
  • $\begingroup$ (Finally getting back to this) You are mixing […] model selection and regularization. – Okay, but what is the difference, in particular when it comes to what problem they solve? Most of the literature I found so far either focuses on one of these, but the illustrative problems seems to fall in the same class to me. In some literature, e.g., van Erp et al. (2019) shrinkage priors and BIC are compared when attacking the same problem. $\endgroup$
    – Wrzlprmft
    Commented Sep 5, 2022 at 13:23
  • $\begingroup$ What you are describing sounds like a research project that can take several weeks or months – Finding out how to attack the general class of problems I describe or how to do it for a specific example? In the latter case, is there any answer to the former? $\endgroup$
    – Wrzlprmft
    Commented Sep 5, 2022 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.