12
$\begingroup$

In most of the examples I've seen so far of neural networks, the network is used for classification and the nodes are transformed with a sigmoid function . However, I would like to use a neural network to output a continuous real value (realistically the output would usually be in the range of -5 to +5).

My questions are:

1. Should I still scale the input features using feature scaling? What range?
2. What transformation function should I use in place of the sigmoid?

I'm looking to initially implement it PyBrain which describes these layer types.

So I'm thinking that I should have 3 layers to start (an input, hidden, and output layer) that are all linear layers? Is that a reasonable way? Or alternatively could I "stretch" the sigmoid function over the range -5 to 5?

$\endgroup$
  • 1
    $\begingroup$ Sure you can use a sigmoid $[-\infty, \infty] \mapsto [-5, 5]$. E.g. start from the logistic function, multiply by 10, subtract 5... $\endgroup$ – cbeleites Apr 10 '12 at 13:13
  • $\begingroup$ Is there a particular reason you're avoiding using two hidden layers? That would seem to be the easiest way to accomplish getting real-valued continuous output from a neural network. "Any function can be approximated to arbitrary accuracy by a network with two hidden layers" (mentioned in notes from the Mitchell machine learning text slide 26: cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/mlbook/ch4.pdf ) $\endgroup$ – Chris Simokat Apr 16 '12 at 1:45
  • $\begingroup$ @ChrisSimokat: No, but most of what I have read so far suggests a single hidden layer as a reasonable starting point. Can a single hidden layer network not approximate any function? $\endgroup$ – User Apr 17 '12 at 5:30
  • 1
    $\begingroup$ @ChrisSimokat: Maybe I'm missing something but I thought single hidden layer does not equal "single layer perceptron", no? $\endgroup$ – User Apr 18 '12 at 20:58
  • 1
    $\begingroup$ No you're not missing anything I just apparently wasn't reading closely enough sorry about that. $\endgroup$ – Chris Simokat Apr 19 '12 at 21:47
11
$\begingroup$

1. Should I still scale the input features using feature scaling? What range?

Scaling does not make anything worse. Read this answer from Sarle's neural network FAQ: Subject: Should I normalize/standardize/rescale the data? .

2. What transformation function should I use in place of the sigmoid?

You could use logistic sigmoid or tanh as activation function. That doesn't matter. You don't have to change the learning algorithm. You just have to scale the outputs of your training set down to the range of the output layer activation function ($[0,1]$ or $[-1,1]$) and when you trained your network, you have to scale the output of your network to $[-5,5]$. You really don't have to change anything else.

$\endgroup$
  • $\begingroup$ What is the correct way to scale the neural network output to the range [-5,5]? $\endgroup$ – User Apr 15 '12 at 11:57
  • 3
    $\begingroup$ To scale element $e \in [a,b]$ to an interval $[c,d]$ you have to calculate $\frac{e-a}{b-a} \cdot (d-c)+c$. $\endgroup$ – alfa Apr 15 '12 at 15:01
  • 1
    $\begingroup$ But since sigmoid is non-linear, with uniform distribution sampling the value of sigmoid we would probably get something close to 1 or close to 0. Which means we have to learn our network to pick values in the middle more carefully. Is sigmoid+scaling really a good choice to go for? $\endgroup$ – Smarty77 May 23 '18 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.