Can the BIC be used in neural nets despite its large no. of parameters? I wonder if I can calculate a BIC value ($BIC=-2\log L(\hat{\theta}) + \log n \cdot d$) for a neural net. The number of parameters is large, $d \approx 20,000$ and I only have $n=700$ samples. Here, https://en.wikipedia.org/wiki/Bayesian_information_criterion, they say that the BIC is only for models with $n>>d$, but I couldn't find another source confirming this statement. 
Does anybody knows something? I thought that they are maybe referring to the regression context, where $d-2$ is also similar to the number of covariates. But this actually does not matter for other regression methods..
 A: BIC (and similarly AIC) assesses a model on the same data, on which your model was trained. As a result they are pretty terrible criteria for models that are massively over-parameterized and somewhat regularized (e.g. drop-out and using a NN in layers instead of densely connected; for some specific types of regularization such as in hierarchical models there are specific solutions like DIC). Hopefully, in a sense your effective number of parameters is much smaller than the actual number of parameters of the neural network (due to regularization, as well as other factors; see e.g. the lottery ticket hypothesis/pruning techniques). It only gets "worse", once you do things like data augmentation (what is $n$ in that case - presumably larger than your number of original records?!) or transfer learning (which in a sense similarly reduces the effective number of parameters like having a Bayesian prior).
Bottom line: you will usually not have the slightest clue as to how badly your model might be overfitting (or not) and having the same number of trainable parameters in different architectures is usually not at all the same thing.
To get a real idea of what is going on and pick a model/decide how to average them, you really have to look at a validation set and/or do cross-validation (while evaluating on a hold-out test set you do not touch while doing that).
