# Choosing cost function for a neural network continuous forecast

I´m using Octave to build a Neural Network regression (3 layers nn, using tanh as activation function, fmincg as optimizer and continuous output). The purpose of the model is to forecast demand for some products, based on multiple variables (past demand of the products, past stock levels, pendent orders, etc.). I'm normalizing all inputs and output.

The quality of the forecast is finally evalued with SFE (statistical forecast error), which I defined in Octave as: $$SFE=\frac{\sum_{i=1}^{m}abs(Y_m-Forecast_m)}{\sum_{i=1}^{m}Forecast_m}$$. Despite this, I'm training the model as a "traditional" regression, using a sum of square errors as cost function (J): $$J=\frac{\sum_{i=1}^{m}(Y_m-Forecast_m)^2}{2m}$$. I think this is not correct, as learning should be made towards the real objective (SFE) and J is over-estimating the impact of bigger deviations. But as abs() has no derivative, I don't realize how SFE function could work under a back-prop training for the nn.

How should the cost function (J) be defined so as to train the model towards SFE? I'd like to use SFE itself, but I don't find it possible as I have to compute the gradient in order for the optimizer to work (fmincg for Octave) and abs() is not differentiable at 0. The answer should cover this aspect for the optimizer to work.

-

It might be detrimental if you use non linear conjugate gradient. Stochastic gradient will have no problem with it. Instead of the absolute value, you can however use $\sqrt{x^2 + \epsilon}$ as an approximation, with a rather small $\epsilon$, e.g. 1e-4. – bayerj Dec 28 '12 at 19:57