Prediction error in Neural Networks

Question

With Random Forests, one can estimate the prediction error using out-of-bag simulations. So for every sample in the training/test-set, one can estimate the predictive uncertainty.

1. What would be the equivalent way to measure predictions error in Neural Nets?
2. Does the oob-error quantify epistemic (model) of aleatoric (data) uncertainty?
3. Can MC dropout here be seen as similar to oob-error?

Answer 1

1. Out-of-bag analysis is possible with every ensemble method based on bootstrapping your data. Thus, nothing keeps you from training multiple networks, each on a bootstrap sample of your entire training sample. For each NN, evaluate prediction accuracy on those training observations not chosen for this particular NN's training set.

   I am not aware of any software that implements this approach, but it should be a straightforward question of wrapping NN training in a loop and bootstrapping your data. (And of using much more training time of course, since the one performance boost of RFs, selecting a small number of allowed splitting predictors, is not available to you. But the loops are trivially parallelizable.)

   I don't think there is any other obvious equivalent way to assess NN prediction error analogously to OOB. (Both that the RF approach can be applied straightforwardly and that there is no other obvious analogue applies to any modeling approach, whether linear regression, trees or NNs.)

2. Both. Note that each tree in the RF will be different, because it is grown from different data. The different trees are your different models, so the forest reflects model uncertainty. And also data uncertainty or variance, through the bootstrap.

3. It would be useful if you could explain what "dropout" is, instead of asking us to read a linked paper. Especially since links can rot over time (yes, even links to arXiv).