2
$\begingroup$

I've been looking into ways to estimate uncertainty for regression tasks on neural networks. One of the obvious options is ensemble modeling. Consider an ensemble of neural networks that all have identical architecture. I would assume that the variation that comes from initialization and batch shuffling would be sufficient to estimate uncertainty without needing variation in the training set. I've seen some sources that suggest adding variation in the training set through bootstrap aggregation (bagging). My intuition tells me that each ensemble member from the bagged ensemble will perform poorer with the added variation in the training set but my uncertainty estimate will probably be more conservative. I'd also expect point estimates from the bagged ensemble to have more error than point estimates from the ensemble without resampling, though I suppose that might be domain and data dependent.

Does anyone here have experience with this and could you tell me if my intuition is right or wrong? I'm specifically interested in ensemble neural networks with and without the variation that comes with resampling of the training set for each network. Other techniques to estimate uncertainty such as using BNNs are a separate topic.

EDIT: I had another thought after writing this. Assume that prior to training we've with held a test set. Then we split the training data into a training set and a validation set where the validation set is used to determine the best model (best model is the model that had the smallest validation loss). Instead of resampling via bootstrap, why not resample a different random split for training/validation? This helps to make sure the validation data is not completely wasted since it is likely some of the validation data for one model will be used as training data for another model. Typically that data is discarded when training a single neural network this way. I'd assume this ensemble would generalize better than a bagged ensemble since not only will each model have more unique samples, but the ensemble overall will also be trained using more data. Or am I missing something?

My intuition is still telling me bagging for neural networks is not a good thing to do. The attached paper I found seems to agree with my intuition. https://www.gatsby.ucl.ac.uk/~balaji/why_arent_bootstrapped_neural_networks_better.pdf

$\endgroup$
3
  • $\begingroup$ Take the extreme case where all of your estimators have 0 loss in their datasets, so they perfectly overfit. If you don't do bagging, then even though they all started from different initialization, they all agree on the points in the dataset $\endgroup$
    – Alberto
    Commented Sep 10, 2023 at 12:37
  • $\begingroup$ I'm not quite following what motivates you to consider this case? I believe it illustrates that overfitting can lead to cases where your estimated uncertainty on predictions for the training set is zero. But it seems unreasonable to think this would be the case for the validation or test sets. I would expect a very large variance in outputs on either validation or test data for this collection of severely overfit models. $\endgroup$
    – noNameTed
    Commented Sep 11, 2023 at 15:19
  • $\begingroup$ I should add that I'm assuming the models above aren't all identical in my comment above (more than one set of parameters can fit the data perfectly). $\endgroup$
    – noNameTed
    Commented Sep 11, 2023 at 15:28

1 Answer 1

0
$\begingroup$

There are many methods in the literature for uncertainty estimation. Ensembles are one standard category of methods. So, yes, training an ensemble of models and then using the distribution of predictions to estimate uncertainty is a very reasonable approach; it is one of the standard methods that are close to the state of the art.

Is it better to use bagging or not, when training this ensemble, if each model is a complex neural network? I don't know. I haven't seen papers that have studied that. In the study of the literature that I've done on uncertainty estimation for neural networks, roughly speaking, the standard methods train an ensemble of models without bagging. But then, that is computationally expensive, because you have to train dozens of models and at inference time you have to run a forward pass through all of them. So, instead, a standard heuristic/approximation to that is instead we train only a single network, but with dropout. We obtain an ensemble by using different random values for the dropout, but the same weights. The benefit is that then we only have to train a single network, which is more tractable. At inference time, we run that model forward multiple times, with different independent random choices for the dropout in each run.

This approach of obtaining an ensemble of models using dropout is often called the MC Dropout (Monte Carlo dropout) method for uncertainty estimation. Because of the use of dropout, it isn't possible to use bagging. For these reasons, the most standard, widely used method for uncertainty estimation with ensembles, based on the research literature I've read on uncertainty estimation for neural networks, does not use bagging.

Does that mean bagging is worse than not using bagging? Not necessarily. I can't recall seeing any paper that studied that. I don't think it's a priori obvious whether bagging would be better. It could be better, if it captures a more accurate estimate of the variability in the prediction. It also could be worse, for the reasons you mention, of having less training data for each model. But it's also possible that those reasons might not apply, if there is sufficient training data that reducing the size of the training data by 2x does not harm the quality of the model too much. It's hard to know.

As with most questions about neural networks, the only way to find out whether bagging is better or worse would be to conduct experiments. I suspect those experiments might well depend on the particular datasets you are using.

So if you really care about it, implement both and benchmark them to see which does better in your experiments.

But if you want to be pragmatic, don't bother. You can skip bagging and know that what you're doing is consistent with a standard scheme in the research literature, thus gaining some assurance that you are doing something reasonable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.