I am trying to predict for count which ranges from 0 onwards as a regression problem using NN. Can I add sigmoid, tanh or relu activation function to the hidden layers and no activation function to the final layer?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You should at least add some sort of activation to the hidden layers, otherwise no matter how many layers you use, it'll act as if the network has a single layer. Let's say you have an hidden layer, with weight and bias $W_h,b_h$, and your output layer is $W_o,b_o$. And, let your input be $x$; then the output is $$o=W_o(W_hx+b_h)+b_o=\underbrace{W_oW_h}_Wx+\underbrace{W_ob_h+b_o}_b=Wx+b$$ This means the SGD tries to learn $W_o,W_h,b_o,b_h$ but it is the same as learning $W,b$ by a single layer NN.
