# Standard Deviation in Neural Network Regression

I have constructed a Neural Network to predict a function of i inputs. ie:

\begin{align*} y = f(x_1, x_2 .. x_i) \end{align*}

The network is working really well and gives me a good approximation of the mean at each point and extrapolates perfectly to the points where I have no data.

Now I believe the true y to be normally distributed around the output y (mean) for each input X. I'm trying to think of a sane way to get the standard deviation here. If I calculate the standard deviation from all matching X inputs and then feed it into a new neural network will that do the trick? I'm worried that leaves me with a severely reduced dataset and the calculation is not as reliable.

With all that dealing with errors in the back propogation step it seems like there should be some way to extract a form of variance...or is that a pipe dream? :D

• Are you talking about bootstrapping? Oct 19, 2017 at 9:40
• Ah, so that's the word I was looking for. Oct 19, 2017 at 9:46

First of all you want to get standard deviations that says something about test errors not training errors. There are different aproaches to solve the problem.

Ensampling/bootstrapping: multiple different splits of your training data and get out of bag estimates for each split. Then for each observation calculate the standard error of th prediction. For test data just calculate the standard errors for all the estimates. The mean of the estimates are by the way a better model than any single model. So this can improve your prediction error.

Modeling the standard error directly: Make a neural network that outputs the (log of the) standard error of the prediction given an input trained on your validation errors. This is then optimized using MLE and should be pretty straight forward. Just train a network with the following objective and the residuals as the targets (logsigma(X) is a neural network outputing a scalar from negative infinity to infinity).

$obj=\sum-1/2*logsigma(X_i)-\frac{residual_i^2}{2*exp(logsigma(X_i)*2)}$

Pros and cons: The ensampling bootstrapping is well tested, but it gives you the standard derivation of the estimate, not the expected standard error of the observation. Modeling the standard error directly is not as well accepted, but it gives you unbiased estimates of the standard error of your residuals for each observation. If you have time and the courage I would try the latter one. You can off course make both, so you make an ensemble of models and make a neural network trained on the out of bag residuals.

• Ok let's start with bootstrapping as there is a lot more info out there that I can readily understand! So as far as I can tell I need to fit a neural network to a number of bootstrapped data sets, then for any X input I find the mean of all the neural net estimates and the error at that X as the average distance from this mean? EDIT: Thanks by the way! Oct 19, 2017 at 11:49
• If this is the right way to do it, presumably the more bootstrapped samples I have, the more neural nets I need to train. If my current single neural net takes 1hr to fit, I could be looking at a long wait... Oct 19, 2017 at 12:01
• Actually that second option is beginning to make sense, and I only need train 2 neural nets (unless I combine methods). I'm not sure I understand the objective function but if it gives me the expected residuals then this is perfect. Oct 19, 2017 at 12:29
• Is there a common name for the objective function you have given above? Is that was MLE is? What does that stand for?! Oct 20, 2017 at 8:49
• it is just derived from the standard regression criteria assuming the residuals are normally distributed with the variance defined by a neural network. So instead of minimizing the negative log likelihood with respect to the conditional mean we are minimizing it with respect to the conditional variance. Oct 21, 2017 at 15:57