Skip to main content
Cross Validated

Return to Question

Notice removed Draw attention by Wrzlprmft
Bounty Ended with Tim's answer chosen by Wrzlprmft
Notice added Draw attention by Wrzlprmft
Bounty Started worth 50 reputation by Wrzlprmft
deleted 11 characters in body
Source Link
Wrzlprmft
Wrzlprmft
  • 2.4k
  • 1
  • 20
  • 38

How to incorporate prior probabilitiesuse priors on the parameter number of parameters with an information criterion (AIC, BIC, …)?

Example

The example is made up because I hope that it’s more accessible than my actual problem.

I want to determine the number of planets of a star using data of said observable. I have:

  • data for some astronomical observable of that star, e.g., an intensity time series,
  • a model that describes the observable with the sizes of the star’s planets as parameters,
  • goodness of fits / Bayes factors for the planet sizes (parameter values) that best explain the data for a given number of planets,
  • prior data on how likely a given number of planets is,
  • no prior data on planet sizes.

The model is such that for an infinite number of planets, it can explain any data. Also, a planet with size zero is equivalent to no planet.

Question

If I have priors on the number of parameters itself, how do I properly penalise additional parameters to avoid overfitting? In the example, how do I penalise an extra planet? I am primarily skeptical as to whether my prior information on the number of planets already covers any extra penalty imposed by an information criterion such as the Akaike or Bayesian information criterion.

Intuitively I would say that it isn’t covered and thus I would use an information criterion with the product of Bayes factor and the known priors for the number of planets as likelihood. My prime rationale for this is that in absence of prior information on the number of planets, i.e., equal priors, I clearly need to have an additional penalty lest I arrive at a model with an infinite number of planets.

On the other hand, I was thinking: What if the ratio between the prior probabilities of two and one and no planet was higher than the parameter penalty imposed by the information criterion? For example, in case of the AIC, what if two planetsone planet were more probable than oneno planet by a factor $e$ (from prior insights)? In that case, my approach would always favour two planets over one planet over none – unless I implement a penalty, lower prior, or similar for very small planets.

How to incorporate prior probabilities on the number of parameters with an information criterion?

Example

The example is made up because I hope that it’s more accessible than my actual problem.

I want to determine the number of planets of a star using data of said observable. I have:

  • data for some astronomical observable of that star, e.g., an intensity time series,
  • a model that describes the observable with the sizes of the star’s planets as parameters,
  • goodness of fits / Bayes factors for the planet sizes (parameter values) that best explain the data for a given number of planets,
  • prior data on how likely a given number of planets is,
  • no prior data on planet sizes.

The model is such that for an infinite number of planets, it can explain any data. Also, a planet with size zero is equivalent to no planet.

Question

If I have priors on the number of parameters itself, how do I properly penalise additional parameters to avoid overfitting? In the example, how do I penalise an extra planet? I am primarily skeptical as to whether my prior information on the number of planets already covers any extra penalty imposed by an information criterion such as the Akaike or Bayesian information criterion.

Intuitively I would say that it isn’t covered and thus I would use an information criterion with the product of Bayes factor and the known priors for the number of planets as likelihood. My prime rationale for this is that in absence of prior information on the number of planets, i.e., equal priors, I clearly need to have an additional penalty lest I arrive at a model with an infinite number of planets.

On the other hand, I was thinking: What if the ratio between the prior probabilities of two and one planet was higher than the parameter penalty imposed by the information criterion? For example, in case of the AIC, what if two planets were more probable than one planet by a factor $e$? In that case, my approach would always favour two planets over one – unless I implement a penalty, lower prior, or similar for very small planets.

How to use priors on the parameter number with an information criterion (AIC, BIC, …)?

Example

The example is made up because I hope that it’s more accessible than my actual problem.

I want to determine the number of planets of a star. I have:

  • data for some astronomical observable of that star, e.g., an intensity time series,
  • a model that describes the observable with the sizes of the star’s planets as parameters,
  • goodness of fits / Bayes factors for the planet sizes (parameter values) that best explain the data for a given number of planets,
  • prior data on how likely a given number of planets is,
  • no prior data on planet sizes.

The model is such that for an infinite number of planets, it can explain any data. Also, a planet with size zero is equivalent to no planet.

Question

If I have priors on the number of parameters itself, how do I properly penalise additional parameters to avoid overfitting? In the example, how do I penalise an extra planet? I am primarily skeptical as to whether my prior information on the number of planets already covers any extra penalty imposed by an information criterion such as the Akaike or Bayesian information criterion.

Intuitively I would say that it isn’t covered and thus I would use an information criterion with the product of Bayes factor and the known priors for the number of planets as likelihood. My prime rationale for this is that in absence of prior information on the number of planets, i.e., equal priors, I clearly need to have an additional penalty lest I arrive at a model with an infinite number of planets.

On the other hand, I was thinking: What if the ratio between the prior probabilities of one and no planet was higher than the parameter penalty imposed by the information criterion? For example, in case of the AIC, what if one planet were more probable than no planet by a factor $e$ (from prior insights)? In that case, my approach would always favour one planet over none – unless I implement a penalty, lower prior, or similar for very small planets.

Source Link
Wrzlprmft
Wrzlprmft
  • 2.4k
  • 1
  • 20
  • 38

How to incorporate prior probabilities on the number of parameters with an information criterion?

Example

The example is made up because I hope that it’s more accessible than my actual problem.

I want to determine the number of planets of a star using data of said observable. I have:

  • data for some astronomical observable of that star, e.g., an intensity time series,
  • a model that describes the observable with the sizes of the star’s planets as parameters,
  • goodness of fits / Bayes factors for the planet sizes (parameter values) that best explain the data for a given number of planets,
  • prior data on how likely a given number of planets is,
  • no prior data on planet sizes.

The model is such that for an infinite number of planets, it can explain any data. Also, a planet with size zero is equivalent to no planet.

Question

If I have priors on the number of parameters itself, how do I properly penalise additional parameters to avoid overfitting? In the example, how do I penalise an extra planet? I am primarily skeptical as to whether my prior information on the number of planets already covers any extra penalty imposed by an information criterion such as the Akaike or Bayesian information criterion.

Intuitively I would say that it isn’t covered and thus I would use an information criterion with the product of Bayes factor and the known priors for the number of planets as likelihood. My prime rationale for this is that in absence of prior information on the number of planets, i.e., equal priors, I clearly need to have an additional penalty lest I arrive at a model with an infinite number of planets.

On the other hand, I was thinking: What if the ratio between the prior probabilities of two and one planet was higher than the parameter penalty imposed by the information criterion? For example, in case of the AIC, what if two planets were more probable than one planet by a factor $e$? In that case, my approach would always favour two planets over one – unless I implement a penalty, lower prior, or similar for very small planets.