Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite

I want to approximate a non-linear function with a limited value range by an artificial neural network (feed forward, back propagation). Most tools and literature availabe suggest linear functions for the output neurons when doing regressions. However, I know a priori that my goal function is of limited range, therefore is it reasonable to use a function for the output neurons with limited value range, too? To be more explicit: My target function's values are in the range between 0 and 1, but the neural net does predict occasionally values that exceed this range (e.g. -1.3). Can i prevent the net from doing so, and is it reasonable?

Thank you very much, Julian

share|improve this question

5 Answers

up vote 1 down vote

If you use a logistic activation function in the output layer it will restrict the output to the range 0-1 as you require. However if you have a regression problem with a restricted output range the sum-of-squares error metric may not be ideal and maybe a beta noise model might be more appropriate (c.f. beta regression, which IIRC is implemented in an R package but I have never used it myself)

share|improve this answer
I replaced the linear function by a siglog-function. Strangely, the neural net predicts now 0.5 for whatever data inputs. May this be a consequence of still using the sum-of-squares metric? – Julian Feb 28 '11 at 14:02
The use of the sum-of-squares error metric shouldn't be a problem, assymptotically it will give a network that predicts the conditional mean of the data regardless of the distribution and output layer activation function. Can you give some more information about the data, size of the netwok, training algorithm etc? What is the distribution of target values like? Are you using regularisation? – Dikran Marsupial Feb 28 '11 at 14:10
Training algorithm is back propagation. 500 training cycles, learning rate 0.3 and 0.2 momentum. 1 hidden layer with 3 neurons. The data consists of 3 attributes, and one target attribute. 20000 samples. The target attribute is very heavily(!!!) skewed within the range of 0 and 1, the mean is 0.077, SD 0.185. – Julian Feb 28 '11 at 14:29
In that case, it might be worth transforming the target to have a less skewed distribution before training. Is it a problem with many values exactly at zero (like rainfall for example) or are there no special values? – Dikran Marsupial Feb 28 '11 at 15:48
1  
Try a lesser learning rate (say 0.001) and a higher momentum (say 0.99). This will make training take longer but it will also be more robust. – bayerj Mar 1 '11 at 14:34
up vote 0 down vote

If you know an absolute range for the output, but there is no reason to expect it to have the non-linear characteristic of the typical logistic activation function (i.e. a value in the middle is just as likely as a value near 0 or 1), then you can just transform the output by dividing by the absolute maximum. If the minimum were not 0, you could subtract the absolute minimum before dividing by the value (maximum - minimum).

So basically don't try to train the neural network to the raw value, train it to the percentile value (0 for minimum, 1 for maximum).

share|improve this answer
That will not prevent the predictions of the network from going outside the range 0-1. If it is necessary to constrain the outputs of the network to lie within a particular range, a non-linear activation function is unavoidable. – Dikran Marsupial Feb 28 '11 at 13:53
If you mean, it won't go to even 1.01, then you're right. However, if you train to values that are all between 0 and 1, then eventually the network will train to always give values close to that range. You can clip the output to 0 to 1 if you want, while leaving the region in between linear, but in most cases if it gives an output of 1.01 or 1.02 occasionally this is not an issue. – rossdavidh Mar 1 '11 at 13:34
up vote 0 down vote

Would it work to use the linear function and simply cut all values below 0 to 0, and values above 1 to 1?

share|improve this answer
up vote 0 down vote

"Would it work to use the linear function and simply cut all values below 0 to 0, and values above 1 to 1?"

I believe in many cases the cut-off value should be the percentage split of the training data. Eg if your training data has 13% - 0's and 87% - 1's, then the cut-off would be 0.13; For example anything 0.13 and below on the output is 0 and anything 0.14 and above is 1. Obviously there is more uncertainty the closer to the cut-off the output provides. It may also help adjusting the cut-off limits especially where the cost of a mis-classification is high. This link may help a little http://timmanns.blogspot.com/2009/11/building-neural-networks-on-unbalanced.html

share|improve this answer
up vote 0 down vote

I am opposed to cutting values of, since this will lead to an undifferentiable transfer function and your gradient based training algorithm might screw up.

The sigmoid function at the output layer is fine: $\sigma(x) = \frac{1}{1 + e^{-x}}$. It will squash any output to lie within $(0, 1)$. So you can get arbitrarily close to the targets.

However, if you use the squared error you will lose the property of a "matching loss function". When using linear outputs for a squared error, the derivatives of the error reduce to $y - t$ where $y$ is the output and $t$ the corresponding target value. So you have to check your gradients.

I have personally had good results with sigmoids as outputs when I have targets in that range and using sum of squares error anyway.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.