15
$\begingroup$

In most of the examples I've seen so far of neural networks, the network is used for classification and the nodes are transformed with a sigmoid function . However, I would like to use a neural network to output a continuous real value (realistically the output would usually be in the range of -5 to +5).

My questions are:

1. Should I still scale the input features using feature scaling? What range?
2. What transformation function should I use in place of the sigmoid?

I'm looking to initially implement it PyBrain which describes these layer types.

So I'm thinking that I should have 3 layers to start (an input, hidden, and output layer) that are all linear layers? Is that a reasonable way? Or alternatively could I "stretch" the sigmoid function over the range -5 to 5?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Sure you can use a sigmoid $[-\infty, \infty] \mapsto [-5, 5]$. E.g. start from the logistic function, multiply by 10, subtract 5... $\endgroup$
    – cbeleites
    Commented Apr 10, 2012 at 13:13
  • $\begingroup$ Is there a particular reason you're avoiding using two hidden layers? That would seem to be the easiest way to accomplish getting real-valued continuous output from a neural network. "Any function can be approximated to arbitrary accuracy by a network with two hidden layers" (mentioned in notes from the Mitchell machine learning text slide 26: cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf ) $\endgroup$
    – Chris Simokat
    Commented Apr 16, 2012 at 1:45
  • $\begingroup$ @ChrisSimokat: No, but most of what I have read so far suggests a single hidden layer as a reasonable starting point. Can a single hidden layer network not approximate any function? $\endgroup$
    – User
    Commented Apr 17, 2012 at 5:30
  • 1
    $\begingroup$ @ChrisSimokat: Maybe I'm missing something but I thought single hidden layer does not equal "single layer perceptron", no? $\endgroup$
    – User
    Commented Apr 18, 2012 at 20:58
  • 1
    $\begingroup$ No you're not missing anything I just apparently wasn't reading closely enough sorry about that. $\endgroup$
    – Chris Simokat
    Commented Apr 19, 2012 at 21:47

2 Answers 2

Reset to default
13
$\begingroup$

1. Should I still scale the input features using feature scaling? What range?

Scaling does not make anything worse. Read this answer from Sarle's neural network FAQ: Subject: Should I normalize/standardize/rescale the data? .

2. What transformation function should I use in place of the sigmoid?

You could use logistic sigmoid or tanh as activation function. That doesn't matter. You don't have to change the learning algorithm. You just have to scale the outputs of your training set down to the range of the output layer activation function ($[0,1]$ or $[-1,1]$) and when you trained your network, you have to scale the output of your network to $[-5,5]$. You really don't have to change anything else.

Improve this answer
$\endgroup$
3
  • $\begingroup$ What is the correct way to scale the neural network output to the range [-5,5]? $\endgroup$
    – User
    Commented Apr 15, 2012 at 11:57
  • 3
    $\begingroup$ To scale element $e \in [a,b]$ to an interval $[c,d]$ you have to calculate $\frac{e-a}{b-a} \cdot (d-c)+c$. $\endgroup$
    – alfa
    Commented Apr 15, 2012 at 15:01
  • 2
    $\begingroup$ But since sigmoid is non-linear, with uniform distribution sampling the value of sigmoid we would probably get something close to 1 or close to 0. Which means we have to learn our network to pick values in the middle more carefully. Is sigmoid+scaling really a good choice to go for? $\endgroup$
    – Smarty77
    Commented May 23, 2018 at 18:57
0
$\begingroup$

Disclaimer: the approach presented is not feasible for continuous values, but I do believe bears some weight in decision making for the project Smarty77 brings up a good point about utilizing a rescaled sigmoid function. Inherently, the sigmoid function produces a probability, which describes a sampling success rate (ie 95 out of 100 photos with these features are successfully 'dog'). The final outcome described is a binary one, and the training, using 'binary cross-entropy' describes a process of separating diametrically opposed outcomes, which inherently discourages results in the middle-range. The continuum of the output is merely there for scaling based on number of samples (ie a result of 0.9761 means that 9761 out of 10000 samples displaying those or similar triats are 'dog'), but each result itself must still be considered to be binary and not arbitrarily granular. As such, it should not be mistaken for and applied as one would real numbers and may not be applicable here. Though I am not sure of the utilization of the network, I would normalize the output vector w.r.t. itself. This can be done with softmax. This will also require there to be 11 linear outputs (bins) from the network (one for each output -5 to +5), one for each class. It will provide an assurance value for any one 'bin' being the correct answer. This architecture would be trainable with one-hot encoding, with the 1 indicating the correct bin. The result is interpretable then in a manner of ways, like a greedy strategy or probabilistic sampling. However, to recast it into a continuous variable, the assuredness of each index can be used as a weight to place a marker on a number-line (similar to the behavior of the sigmoid unit), but this also highlights the primary issue: if the network is fairly certain the result is -2 or +3, but absolutely certain that it is not anything else, is +1 a viable result? Thank you for your consideration. Good luck on your project.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.