I am having an issue with using neural networks. I started with something simple. I just used nntool with one hidden layer (with one neuron) with linear activation function. For the output also, I used the linear activation function. I just fed my Xs and Ys to the neural network tool and got the results on the testing set.

I compared that with normal ridge function in MATLAB.

I could see that neural network one performed much worse than ridge function. The first reason is that there are lots of negative values in the predictions, when my target is only positive. Ridge regression gave about 800 -ve values while nn gave around 5000 -ve values which totally ruined the accuracy of nntool.

Since nntool can be used to perform linear regression, why nntool is not performing as well as ridge regression?

What about the negative values what's the reason behind it and how to enforce positive values?

If you have a linear activation function, your neural network collapses to a linear model: the composition of a linear function with a linear function remains linear.

Thus, your neural net will not be able to perform better than ridge regression. Indeed, you are optimizing something very similar with stochastic gradient descent which will result in sub optimal solutions very often. Ridge regression, on the other hand, can be solved in closed form and you will always get the best result wrt training error.

If you want to have better results than ridge regression, you will need to use nonlinear activation functions.

  • Yeah I tried with nonlinear activation function and it performed worse than linear one even on the training dataset. As I have said as I started using non linear activation function with more number of neurons in the hidden layer, it kept getting worse and I think the reason is because of larger number of -ve values in the output. How is that even possible – user34790 Sep 4 '13 at 14:16
  • If it is worse than linear, it is an optimization issue. Try to play with parameters such as the learning rate. (if you are using stochastic gradient descent... if you are using something more sophisticated, try using SGD). – bayerj Sep 5 '13 at 20:27

There are many reasons why a neural network could underperform ridge regression. First, simply because ridge regression and a single node NN both return a linear fit doesn't mean that they will return the same one. In fact, they hardly ever will, unless your ridge parameter is zero.

If the NN's optimization function is able to find a global minimum (which with a single neuron, it should unless you've misparametrized it badly), then it will actually be doing the equivalent of an ordinary non-penalized least squares regression. So a natural reason why ridge regression might work better (in terms of performance on the test set) is that a ridge regression shrinks coefficients in order to avoid overfitting. Try doing an ordinary linear regression and compare the results to the NN.

As for the negative values, there is no way to prevent a linear fit from taking on negative values. If your $Y$ values are always positive, one simple thing to try is fitting $\text{log}(Y)$ instead of $Y$, and then exponentiate the results.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.