4
$\begingroup$

I have constructed a Neural Network to predict a function of i inputs. ie:

\begin{align*} y = f(x_1, x_2 .. x_i) \end{align*}

The network is working really well and gives me a good approximation of the mean at each point and extrapolates perfectly to the points where I have no data.

Now I believe the true y to be normally distributed around the output y (mean) for each input X. I'm trying to think of a sane way to get the standard deviation here. If I calculate the standard deviation from all matching X inputs and then feed it into a new neural network will that do the trick? I'm worried that leaves me with a severely reduced dataset and the calculation is not as reliable.

With all that dealing with errors in the back propogation step it seems like there should be some way to extract a form of variance...or is that a pipe dream? :D

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Are you talking about bootstrapping? $\endgroup$ – SmallChess Oct 19 '17 at 9:40
  • $\begingroup$ Ah, so that's the word I was looking for. $\endgroup$ – Duane Allman Oct 19 '17 at 9:46
4
$\begingroup$

First of all you want to get standard deviations that says something about test errors not training errors. There are different aproaches to solve the problem.

Ensampling/bootstrapping: multiple different splits of your training data and get out of bag estimates for each split. Then for each observation calculate the standard error of th prediction. For test data just calculate the standard errors for all the estimates. The mean of the estimates are by the way a better model than any single model. So this can improve your prediction error.

Modeling the standard error directly: Make a neural network that outputs the (log of the) standard error of the prediction given an input trained on your validation errors. This is then optimized using MLE and should be pretty straight forward. Just train a network with the following objective and the residuals as the targets (logsigma(X) is a neural network outputing a scalar from negative infinity to infinity).

$obj=\sum-1/2*logsigma(X_i)-\frac{residual_i^2}{2*exp(logsigma(X_i)*2)}$

Pros and cons: The ensampling bootstrapping is well tested, but it gives you the standard derivation of the estimate, not the expected standard error of the observation. Modeling the standard error directly is not as well accepted, but it gives you unbiased estimates of the standard error of your residuals for each observation. If you have time and the courage I would try the latter one. You can off course make both, so you make an ensemble of models and make a neural network trained on the out of bag residuals.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Ok let's start with bootstrapping as there is a lot more info out there that I can readily understand! So as far as I can tell I need to fit a neural network to a number of bootstrapped data sets, then for any X input I find the mean of all the neural net estimates and the error at that X as the average distance from this mean? EDIT: Thanks by the way! $\endgroup$ – Duane Allman Oct 19 '17 at 11:49
  • $\begingroup$ If this is the right way to do it, presumably the more bootstrapped samples I have, the more neural nets I need to train. If my current single neural net takes 1hr to fit, I could be looking at a long wait... $\endgroup$ – Duane Allman Oct 19 '17 at 12:01
  • $\begingroup$ Actually that second option is beginning to make sense, and I only need train 2 neural nets (unless I combine methods). I'm not sure I understand the objective function but if it gives me the expected residuals then this is perfect. $\endgroup$ – Duane Allman Oct 19 '17 at 12:29
  • $\begingroup$ Is there a common name for the objective function you have given above? Is that was MLE is? What does that stand for?! $\endgroup$ – Duane Allman Oct 20 '17 at 8:49
  • $\begingroup$ it is just derived from the standard regression criteria assuming the residuals are normally distributed with the variance defined by a neural network. So instead of minimizing the negative log likelihood with respect to the conditional mean we are minimizing it with respect to the conditional variance. $\endgroup$ – Peter Mølgaard Pallesen Oct 21 '17 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.