# In Bayesian Information Criterion (BIC), why does having bigger n get penalized?

The Bayesian Information Criterion (BIC) is calculated with:

$$\text{BIC} = \frac{1}{n \hat{\delta}^2} \Big(\text{RSS} + \ln(n) d \hat{\delta}^2 \Big)$$

where RSS is residual sum of squares and delta squared is estimate of the variance of the error associated with each response measurement.

Q1. Why does having more sample size get penalized, when usually having bigger data sample size is always better than having few?

I have learned that having more sample data size is always better. For example, if you have more data samples, you will have smaller standard error, narrower confidence interval and smaller standard deviation.

But according to this BIC's formula, the statistical model with more sample data would get penalized, which means having less chance to get selected. It gets more obvious when BIC is compared to AIC. As AIC uses 2 instead of ln(n) in its formula, if the sample size n of the model is bigger than 7, that model has less chance to get selected when we use BIC as a way of choosing the optimal model. Why would the creator of BIC want to penalize the model with bigger number of sample size n?

Q2. Why does my textbook 'An Introduction to Statistical Learning' change the meaning of n to 'variable', when we have d, which stands for the number of predictors in the statistical model?

My books says as follows about BIC.

Notice that BIC replaces the $$2 d \hat{\delta}^2$$ used by Cp with a $$\ln(n) d \hat{\delta}^2$$ term, where n is the number of observations. Since ln(n) >2 for any n>7, the BIC statistics generally places a heavier penalty on models with many variables, and hence results in the selection of smaller models than Cp. (p 212)

I cannot guess why the author of this book changed the meaning of n, from 'the number of observations (sample data points) to 'the number of variables'. Don't we already have the variable d, which shows the number of predictor varaibles plus intercept?

I would deeply appreciate if anyone here can answer my two questions. Thank you very much for reading!

First of all, you select the model that has the minimum value when using such criteria, therefore n has the opposite effect than you wrote down since increase in n alone will decrease the value.

Secondly, the information criteria is used to select between different models, not to select between different samples. The reason for these criteria to be used is the fact that adding more parameters will always increase the fit however it does not necessarily mean that the model is better due parsimony and degrees of freedom concerns in academy and overfitting concerns in practice.

A criterion such as BIC will be used to compare models that have different variables, where n will be the same. Therefore n is not there to penalize or favor the sample size. I am guessing it is there to normalize RSS, since RSS will increase indefinitely with n. On contrast, adding more parameters is penalized as it increases the value of the criteria.

• Thank you for your answer. You clarified the fact that we are comparing the model with the "same sample size", but with "different number of parameters". However, I am still curious about the reason why BIC penalizes a sample with the same number of parameters more heavily when the sample has bigger number of sample size? To elaborate on this, let's assume we have a two problems to build a statiatical model on. The first case has 20 sample size whle the second one has 200 sample size. Why does BIC penalize using more parameters in the second case, which has bigger number of sample size? Commented Apr 7, 2018 at 23:08
• No, you are still comparing BIC values with different sample sizes. Absolute BIC value by itself do not have much meaning. It is meant to be a model selection tool for different models. It does penalize parameters more than AIC does, though. Commented Apr 23, 2018 at 10:02
• If however, you are pointing out the change in difference between AIC and BIC values as sample size changes, then you are right that it changes with the sample size. AIC is asymptotically not efficient where BIC is. For large sample sizes, BIC might be more appropriate criteria for that reason. Commented Apr 23, 2018 at 10:10
• editing previous comment: asymptotically consistent. (not asymptotically efficient.) Commented May 7, 2018 at 10:33

$C_p$ (and AIC) penalize each parameter with a factor of 2. BIC penalizes each parameter with a factor $\ln(n)$ which, for $n>7$ is greater than two, as stated in the paragraph you quote. Therefore, BIC places a greater penalty on each parameter and will tend to select more parsimonious models than AIC or $C_p$.

As you have been told, these criteria are meant to compare models fitted to the same sample with different numbers of parameters. All boils down to "correcting" a goodness-of-fit statistic (minus log likelihood, sum of residuals, or functions thereof) with the "cost" of the parameters fitted. BIC simply places (for $n>7$) a higher price tag to each parameter, and one that increases slightly with $n$.

• Thank you for your answer. You clarified the fact that we are comparing the model with the "same sample size", but with "different number of parameters". However, I am still curious about the reason why BIC penalizes a sample with the same number of parameters more heavily when the sample has bigger number of sample size? To elaborate on this, let's assume we have a two problems to build a statiatical model on. The first case has 20 sample size whle the second one has 200 sample size. Why does BIC penalize using more parameters in the second case, which has bigger number of sample size? Commented Apr 7, 2018 at 23:09
• It can be shown that AIC (that is close to $C_p$) "overshoots" the target: tends to give overparameterized models with probability that does not tend to zero as the sample size increases. This is not a fault in AIC, whose aim is to find models with optimal prediction capability. The slightly heavier penalty in BIC can be rationalized as an attempt to avoid the overparameterization AIC is likely to produce. You might want to look the book by Burnham & Anderson for an in-depth discussion of model selection criteria. Commented Apr 9, 2018 at 7:40
• Dang it, I came here because I have the exact same question as Eiffelbear. Why is having more parameters "worse" when you have a higher n? I would think it would make it more okay. Commented Mar 17, 2021 at 0:45
• @JonathanMugan: Having more parameters is not by itself bad, what is bad is to have more parameters than the phenomenon you want to model warrants. AIC is -2loglik + 2p, and the first term is dependent on the sample size: eventually the 2p will become unimportant as compared to -2loglik, hence AIC will likely increase the number of parameters beyond the "correct" number. AIC is known to over-parameterize. BIC (and others) place a larger penalty in the number of parameters to correct for that. Commented Mar 17, 2021 at 9:19

There is a mistake in your inference. BIC does not penalize more data. The actual dependence on n is $$\frac{\ln(n)}{n}$$ which is a monotonically decreasing function for n>2 and thus, decreases (not increases) the penalty when n increases. When compared to AIC which is simply $$\frac{1}{n}$$, the decrease in penalty due to a large n is lesser. In summary, it does decrease the penalty with an increase in n but the amount of decrease is smaller than that in AIC.

Here is a more intuitive answer:

For Q1: As you said, a larger sample size can support more parameters, meaning that some irrelative predictors might be included when we have a larger sample size. Thus, setting a penalty depending on sample size can control this effect. In other words, we want to have a larger penalty for models with a larger sample size.

For Q2: Your textbook does NOT change the meaning of n. n still means the observations. I guess that you were confused by the last sentence. But in fact, it is comparing the penalty term between AIC and BIC. In AIC, the penalty is 2*d, while in BIC, the penalty is ln(n)*d. As long as or sample size is larger than 7, the penalty in BIC is larger than the penalty in AIC.