I want to approximate a non-linear function with a limited value range by an artificial neural network (feed forward, back propagation). Most tools and literature availabe suggest linear functions for the output neurons when doing regressions. However, I know a priori that my goal function is of limited range, therefore is it reasonable to use a function for the output neurons with limited value range, too? To be more explicit: My target function's values are in the range between 0 and 1, but the neural net does predict occasionally values that exceed this range (e.g. -1.3). Can i prevent the net from doing so, and is it reasonable?
$\begingroup$
$\endgroup$
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
I am opposed to cutting values of, since this will lead to an undifferentiable transfer function and your gradient based training algorithm might screw up.
The sigmoid function at the output layer is fine: $\sigma(x) = \frac{1}{1 + e^{-x}}$. It will squash any output to lie within $(0, 1)$. So you can get arbitrarily close to the targets.
However, if you use the squared error you will lose the property of a "matching loss function". When using linear outputs for a squared error, the derivatives of the error reduce to $y - t$ where $y$ is the output and $t$ the corresponding target value. So you have to check your gradients.
I have personally had good results with sigmoids as outputs when I have targets in that range and using sum of squares error anyway.
$\begingroup$
$\endgroup$
5
If you use a logistic activation function in the output layer it will restrict the output to the range 0-1 as you require.
However if you have a regression problem with a restricted output range the sum-of-squares error metric may not be ideal and maybe a beta noise model might be more appropriate (c.f. beta regression, which IIRC is implemented in an R package, but I have never used it myself)
answered Feb 28, 2011 at 13:01
-
$\begingroup$ I replaced the linear function by a siglog-function. Strangely, the neural net predicts now 0.5 for whatever data inputs. May this be a consequence of still using the sum-of-squares metric? $\endgroup$– JulianFeb 28, 2011 at 14:02
-
$\begingroup$ The use of the sum-of-squares error metric shouldn't be a problem, assymptotically it will give a network that predicts the conditional mean of the data regardless of the distribution and output layer activation function. Can you give some more information about the data, size of the netwok, training algorithm etc? What is the distribution of target values like? Are you using regularisation? $\endgroup$ Feb 28, 2011 at 14:10
-
$\begingroup$ Training algorithm is back propagation. 500 training cycles, learning rate 0.3 and 0.2 momentum. 1 hidden layer with 3 neurons. The data consists of 3 attributes, and one target attribute. 20000 samples. The target attribute is very heavily(!!!) skewed within the range of 0 and 1, the mean is 0.077, SD 0.185. $\endgroup$– JulianFeb 28, 2011 at 14:29
-
$\begingroup$ In that case, it might be worth transforming the target to have a less skewed distribution before training. Is it a problem with many values exactly at zero (like rainfall for example) or are there no special values? $\endgroup$ Feb 28, 2011 at 15:48
-
1$\begingroup$ Try a lesser learning rate (say 0.001) and a higher momentum (say 0.99). This will make training take longer but it will also be more robust. $\endgroup$– bayerjMar 1, 2011 at 14:34
$\begingroup$
$\endgroup$
2
If you know an absolute range for the output, but there is no reason to expect it to have the non-linear characteristic of the typical logistic activation function (i.e. a value in the middle is just as likely as a value near 0 or 1), then you can just transform the output by dividing by the absolute maximum. If the minimum were not 0, you could subtract the absolute minimum before dividing by the value (maximum - minimum).
So basically don't try to train the neural network to the raw value, train it to the percentile value (0 for minimum, 1 for maximum).
-
$\begingroup$ That will not prevent the predictions of the network from going outside the range 0-1. If it is necessary to constrain the outputs of the network to lie within a particular range, a non-linear activation function is unavoidable. $\endgroup$ Feb 28, 2011 at 13:53
-
$\begingroup$ If you mean, it won't go to even 1.01, then you're right. However, if you train to values that are all between 0 and 1, then eventually the network will train to always give values close to that range. You can clip the output to 0 to 1 if you want, while leaving the region in between linear, but in most cases if it gives an output of 1.01 or 1.02 occasionally this is not an issue. $\endgroup$ Mar 1, 2011 at 13:34
$\begingroup$
$\endgroup$
"Would it work to use the linear function and simply cut all values below 0 to 0, and values above 1 to 1?"
I believe in many cases the cut-off value should be the percentage split of the training data. Eg if your training data has 13% - 0's and 87% - 1's, then the cut-off would be 0.13; For example anything 0.13 and below on the output is 0 and anything 0.14 and above is 1. Obviously there is more uncertainty the closer to the cut-off the output provides. It may also help adjusting the cut-off limits especially where the cost of a mis-classification is high. This link may help a little http://timmanns.blogspot.com/2009/11/building-neural-networks-on-unbalanced.html