# Can the BIC be used in neural nets despite its large no. of parameters?

I wonder if I can calculate a BIC value ($$BIC=-2\log L(\hat{\theta}) + \log n \cdot d$$) for a neural net. The number of parameters is large, $$d \approx 20,000$$ and I only have $$n=700$$ samples. Here, https://en.wikipedia.org/wiki/Bayesian_information_criterion, they say that the BIC is only for models with $$n>>d$$, but I couldn't find another source confirming this statement.

Does anybody knows something? I thought that they are maybe referring to the regression context, where $$d-2$$ is also similar to the number of covariates. But this actually does not matter for other regression methods..

• There is nothing preventing you from using BIC as a model selection criterion, but the question would be "why would you want to use it?" rather than "why not?" Commented Jun 11, 2019 at 8:29
• So $n<d$ is theoretically no problem? I need the BIC to derive weights for BMA (see my comment below please) Commented Jun 11, 2019 at 12:54

BIC (and similarly AIC) assesses a model on the same data, on which your model was trained. As a result they are pretty terrible criteria for models that are massively over-parameterized and somewhat regularized (e.g. drop-out and using a NN in layers instead of densely connected; for some specific types of regularization such as in hierarchical models there are specific solutions like DIC). Hopefully, in a sense your effective number of parameters is much smaller than the actual number of parameters of the neural network (due to regularization, as well as other factors; see e.g. the lottery ticket hypothesis/pruning techniques). It only gets "worse", once you do things like data augmentation (what is $$n$$ in that case - presumably larger than your number of original records?!) or transfer learning (which in a sense similarly reduces the effective number of parameters like having a Bayesian prior).
• Thank you! Yes, I use dropout, so, if I get you correctly, my parameters are probably less than $20,000$.. but it is still a lot. For clarification, I want to use the approximation of the posterior model probability $$P(M_k|D) = \frac{\exp (-0.5 BIC_k)}{\sum \exp(-0.5 BIC_i)},$$ with $M_k$ being different models, that is e.g. used in Bayes model averaging as the weights. Therefore I want to be sure that it is theoretically okay to calculate the BIC even when $n < d$. Commented Jun 11, 2019 at 12:52
• But can you tell me what exactly is not okay? I mean I know that these weights result in a probability for one of the models and probability almost zero for the others. But I don't know if there is a theoretical problem of this approximation when $n < d$. Commented Jun 11, 2019 at 14:52