# How to use priors on the parameter number with an information criterion (AIC, BIC, …)?

### Example

The example is made up because I hope that it’s more accessible than my actual problem.

I want to determine the number of planets of a star. I have:

• data for some astronomical observable of that star, e.g., an intensity time series,
• a model that describes the observable with the sizes of the star’s planets as parameters,
• goodness of fits / Bayes factors for the planet sizes (parameter values) that best explain the data for a given number of planets,
• prior data on how likely a given number of planets is,
• no prior data on planet sizes.

The model is such that for an infinite number of planets, it can explain any data. Also, a planet with size zero is equivalent to no planet.

### Question

If I have priors on the number of parameters itself, how do I properly penalise additional parameters to avoid overfitting? In the example, how do I penalise an extra planet? I am primarily skeptical as to whether my prior information on the number of planets already covers any extra penalty imposed by an information criterion such as the Akaike or Bayesian information criterion.

Intuitively I would say that it isn’t covered and thus I would use an information criterion with the product of Bayes factor and the known priors for the number of planets as likelihood. My prime rationale for this is that in absence of prior information on the number of planets, i.e., equal priors, I clearly need to have an additional penalty lest I arrive at a model with an infinite number of planets.

On the other hand, I was thinking: What if the ratio between the prior probabilities of one and no planet was higher than the parameter penalty imposed by the information criterion? For example, in case of the AIC, what if one planet were more probable than no planet by a factor $$e$$ (from prior insights)? In that case, my approach would always favour one planet over none – unless I implement a penalty, lower prior, or similar for very small planets.

• I think the vagueness of the example makes it harder to discern the question. I would have thought that more planets would mean more parameters to describe them. As you already have a prior on the number of parameters, if it penalises large models, it is already helping to avoid overfitting, Note also you may want to avoid overfitting by penalising the values of parameters (e.g. regularisation). The idea that complexity is only measured by the number of parameters is a bit past its sell by date. Commented Sep 13, 2022 at 13:21
• @DikranMarsupial: I would have thought that more planets would mean more parameters to describe them. – They do. Each planet comes with one extra parameter, namely its size. (I hoped this was clear from the question.) — As you already have a prior on the number of parameters, if it penalises large models, it is already helping to avoid overfitting – Well, if you so wish, my question is whether this is sufficient or some information criterion needs to be included on top. The example when my prior says that one planet is more likely than none suggests this. Commented Sep 13, 2022 at 13:33
• there is no way of telling whether a prior on the number of parameters is sufficient to avoid overfitting without trying other forms of overfitting avoidance and seeing what happens. The more (correct) prior knowledge you put into a model, the more likely it is to perform well, but if you have enough data your priors become irrelevant - so there is a compromise that also involves data quality and quantity. If you want to avoid ambiguity, then set out the model formally with equations. Commented Sep 13, 2022 at 13:51