3
$\begingroup$

I wonder if I can calculate a BIC value ($BIC=-2\log L(\hat{\theta}) + \log n \cdot d$) for a neural net. The number of parameters is large, $d \approx 20,000$ and I only have $n=700$ samples. Here, https://en.wikipedia.org/wiki/Bayesian_information_criterion, they say that the BIC is only for models with $n>>d$, but I couldn't find another source confirming this statement.

Does anybody knows something? I thought that they are maybe referring to the regression context, where $d-2$ is also similar to the number of covariates. But this actually does not matter for other regression methods..

Improve this question
$\endgroup$
2
  • $\begingroup$ There is nothing preventing you from using BIC as a model selection criterion, but the question would be "why would you want to use it?" rather than "why not?" $\endgroup$
    – David
    Commented Jun 11, 2019 at 8:29
  • $\begingroup$ So $n<d$ is theoretically no problem? I need the BIC to derive weights for BMA (see my comment below please) $\endgroup$
    – msloryg
    Commented Jun 11, 2019 at 12:54

1 Answer 1

Reset to default
3
$\begingroup$

BIC (and similarly AIC) assesses a model on the same data, on which your model was trained. As a result they are pretty terrible criteria for models that are massively over-parameterized and somewhat regularized (e.g. drop-out and using a NN in layers instead of densely connected; for some specific types of regularization such as in hierarchical models there are specific solutions like DIC). Hopefully, in a sense your effective number of parameters is much smaller than the actual number of parameters of the neural network (due to regularization, as well as other factors; see e.g. the lottery ticket hypothesis/pruning techniques). It only gets "worse", once you do things like data augmentation (what is $n$ in that case - presumably larger than your number of original records?!) or transfer learning (which in a sense similarly reduces the effective number of parameters like having a Bayesian prior).

Bottom line: you will usually not have the slightest clue as to how badly your model might be overfitting (or not) and having the same number of trainable parameters in different architectures is usually not at all the same thing.

To get a real idea of what is going on and pick a model/decide how to average them, you really have to look at a validation set and/or do cross-validation (while evaluating on a hold-out test set you do not touch while doing that).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you! Yes, I use dropout, so, if I get you correctly, my parameters are probably less than $20,000$.. but it is still a lot. For clarification, I want to use the approximation of the posterior model probability \begin{equation} P(M_k|D) = \frac{\exp (-0.5 BIC_k)}{\sum \exp(-0.5 BIC_i)}, \end{equation} with $M_k$ being different models, that is e.g. used in Bayes model averaging as the weights. Therefore I want to be sure that it is theoretically okay to calculate the BIC even when $n < d$. $\endgroup$
    – msloryg
    Commented Jun 11, 2019 at 12:52
  • $\begingroup$ It is probably not really okay. I would not trust those model weights. If you want to weight / model average different models, I would base this on what combination performs well during cross-validation (or if you cannot afford it, on a single validation set). $\endgroup$
    – Björn
    Commented Jun 11, 2019 at 13:58
  • $\begingroup$ But can you tell me what exactly is not okay? I mean I know that these weights result in a probability for one of the models and probability almost zero for the others. But I don't know if there is a theoretical problem of this approximation when $n < d$. $\endgroup$
    – msloryg
    Commented Jun 11, 2019 at 14:52
  • $\begingroup$ With d=n models can essentially memorize each data point and give the 'correct' answer back for the training data (and something arbitrary such as 42 for any other input). A much more sensible neural network with d>n might do something much more sensible and perform way better on new data, but will always look worse than a the ridiculous model that memorized the training data. $\endgroup$
    – Björn
    Commented Jun 11, 2019 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.