3
$\begingroup$

I am trying to use neural network to learn a non linear function mapping input to outputs. However, I am having some issues with it. I used tansig activation function for the hidden layers and for the output I used logsig. I scaled the output variables in the range [0 1]. The input variables were standardized to zero mean and unit std. I learn the neural network. Now when I test my NN model on either the train set or a different test set, the outputs are always greater than 0.5. Why is it so? My targets could be anything from 0 to 1. However, my outputs from the model are at least 0.5. Any suggestions what could have happened?

I have around 600 features with 5000 training examples and 10 neurons in the hidden layer and a single output

enter image description here

I tried with tansig activation function for the hidden layers and a linear activation function for the output layers and I got the following. This is somewhat better than before. However, I am a bit surprised why bias is introduced

enter image description here

$\endgroup$
3
$\begingroup$

I can see three potential reasons for that:

  • $logsig$ values abouve $0.5$ simply mean, that sum of activations from the hidden layer is always non-negative. Maybe you forgot to include the bias (bias neuron) in your hidden layer?
  • You standarized your data using some mapping $\phi(x_i)\rightarrow x'_i$. Maybe you trained your network with $(x'_i, y_i)$ and tested on $(x_i,y_i)$ (forgetting about input standarization)?
  • You mapped your outputs to $[0,1]$ and network learned values in $[0.5,1]$, maybe there is some implementation error and you actually provided network with training set of form $(x'_i, logsig(y_i))$ instead of $(x'_i,y_i)$?
$\endgroup$
  • $\begingroup$ Thanks for trying.Definitely the sum of the activations from the hidden layer is always non-negative. But I am wondering why is it so? How to check it $\endgroup$ – user34790 Aug 28 '13 at 5:15
  • $\begingroup$ I gave you three possible reasons, each one needs a check $\endgroup$ – lejlot Aug 28 '13 at 5:18
  • $\begingroup$ The second and third are not true in my case. I am only worried about one $\endgroup$ – user34790 Aug 28 '13 at 5:20
  • $\begingroup$ Then check the manual of library you are using and make sure that the bias neuron is "turned on" (some libraries do not use bias as default option) $\endgroup$ – lejlot Aug 28 '13 at 5:22
  • $\begingroup$ The bias is turned on. When I view the network, I can see it. I am using nntool by the way in matlab. It has the bias turned on $\endgroup$ – user34790 Aug 28 '13 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.