6
$\begingroup$

I am using a neural network with a linear activation function as the output function. My response is continuous and cannot be negative nor can it exceed a certain value. the value is usually between 0 and 1000.

I have used a log response to avoid negative values but When I run the neural network some values will exceed 1000 and be unreasonable since these values are not possible.

What are my options if I want bounded output and at the same time I want to use mini-batch stochastic gradient descent as the optimization function.

I have tried scaling the response between -1 and 1 and using tanh activation function as the output function and also sigmoid as the activation function and scaling the response between 0 and 1. These prevents the predicted values to be outside the bounds 0 and 1000 but does not result in an overall better model in terms of RMSE and MAE.

Is there other models one could try? Is tanh and sigmoid as the activation good shoices for a continuous output?

I'm also wondering if anyone has any suggestions of things to try to improve the neural network with a linear output activation function.

Help would be appreciated.

$\endgroup$
  • 1
    $\begingroup$ What does the distribution of the output look like over [0,1000]? For linear input, the output distribution from a sigmoid/tanh will depend on where the input falls. For example, a narrow input will be transformed almost linearly, but a broad input will produce bunching up around the limits (i.e. peaks in the output histogram at 0 and 1000). $\endgroup$ – GeoMatt22 Sep 5 '16 at 18:16
  • $\begingroup$ The output has a skewed distributions with values often having a large value, does this make it inapropriate to use tanh as the output activation function? $\endgroup$ – user108844 Sep 17 '16 at 8:15
2
$\begingroup$

A trick for bounded output range is to scale the target values between (0,1) and use sigmoid output + binary cross-entropy loss.

This is often used for image data, where all the pixel values are between (0,255).
Say $a=wh+b$ is the activation of the last layer, for sigmoid output + binary cross-entropy loss $$E(a,t')=t'\log\sigma(a)+(1-t')\log(1-\sigma(a)),\quad \frac{\partial E}{\partial a}=\sigma(a)-t'$$ where $t'$ is the scaled target value. The derivative wrt $a$ is just prediction - target, which is somewhat similar to the derivative of using unbounded activations + MSE.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.