Standard Deviation in Neural Network Regression

Question

I have constructed a Neural Network to predict a function of i inputs. ie:

\begin{align*} y = f(x_1, x_2 .. x_i) \end{align*}

The network is working really well and gives me a good approximation of the mean at each point and extrapolates perfectly to the points where I have no data.

Now I believe the true y to be normally distributed around the output y (mean) for each input X. I'm trying to think of a sane way to get the standard deviation here. If I calculate the standard deviation from all matching X inputs and then feed it into a new neural network will that do the trick? I'm worried that leaves me with a severely reduced dataset and the calculation is not as reliable.

With all that dealing with errors in the back propogation step it seems like there should be some way to extract a form of variance...or is that a pipe dream? :D

Answer 1

First of all you want to get standard deviations that says something about test errors not training errors. There are different aproaches to solve the problem.

Ensampling/bootstrapping: multiple different splits of your training data and get out of bag estimates for each split. Then for each observation calculate the standard error of th prediction. For test data just calculate the standard errors for all the estimates. The mean of the estimates are by the way a better model than any single model. So this can improve your prediction error.

Modeling the standard error directly: Make a neural network that outputs the (log of the) standard error of the prediction given an input trained on your validation errors. This is then optimized using MLE and should be pretty straight forward. Just train a network with the following objective and the residuals as the targets (logsigma(X) is a neural network outputing a scalar from negative infinity to infinity).

$obj=\sum-1/2*logsigma(X_i)-\frac{residual_i^2}{2*exp(logsigma(X_i)*2)}$

Pros and cons: The ensampling bootstrapping is well tested, but it gives you the standard derivation of the estimate, not the expected standard error of the observation. Modeling the standard error directly is not as well accepted, but it gives you unbiased estimates of the standard error of your residuals for each observation. If you have time and the courage I would try the latter one. You can off course make both, so you make an ensemble of models and make a neural network trained on the out of bag residuals.