# Multivariate linear regression vs neural network?

It seems that it is possible to get similar results to a neural network with a multivariate linear regression in some cases, and multivariate linear regression is super fast and easy.

Under what circumstances can neural networks give better results than multivariate linear regression?

-

Neural networks can in principle model nonlinearities automatically, which you would need to explicitly model using transformations (splines etc.) in linear regression.

The caveat: the temptation to overfit can be (even) stronger in neural networks than in regression, since adding hidden layers or neurons looks harmless. So be extra careful to look at out-of-sample prediction performance.

-
Ok. I guess a question in my mind is, to what extent can I replicate similar behavior by augmenting my input data with quadratic and cubic terms? –  Hugh Perkins Oct 27 '12 at 9:32
Actually, you can probably approximate NNs with appropriately transformed regressors in a linear regression as closely as you want (and vice versa). Better practice than quadratics and cubics are splines, though - I heartily recommend Harrell's textbook "Regression Modeling Strategies". –  Stephan Kolassa Oct 27 '12 at 10:42
Ok. Is it reasonable to assume that training time will be faster for linear regression on transformed data, or will the training times be approximately similar? Will the solution for the linear regression on transformed data have a single global maximum, or will it have lots of local minimum as for neural networks? (Edit: I guess no matter how the inputs are transformed, the solution to the linear regression is just the pseudoinverse of the design matrix multiplied by something-something and therefore is always either unique or singular?) –  Hugh Perkins Oct 27 '12 at 11:22
Training times will of course depend on input dimensions (few/many observations, few/many predictors). Linear regression involves a single (pseudo-)inverse (yes, uniqueness/singularity even with transformed regressors holds), whereas NNs are typically trained in an iterative way, but iterations don't involve matrix inversions, so each iteration is faster - you typically stop the training based on some criterion designed to stop you from overfitting. –  Stephan Kolassa Oct 27 '12 at 11:35
Hi Stephan, great heads up on the splines. They are awesome. Thanks! For the training time, yes, I see now, a poly(2) or poly(3) or two doesnt affect the matrix dimensions too much, but throwing in a few poly(20)s and stuff, or rather, a few splines like this, soon slows things down considerably! –  Hugh Perkins Oct 28 '12 at 4:30

You mention linear regression. This is related to logistic regression, which has a similar fast optimization algorithm. If you have bounds on the target values, such as with a classification problem, you can view logistic regression as a generalization of linear regression.

Neural networks are strictly more general than logistic regression on the original inputs, since that corresponds to a skip-layer network (with connections directly connecting the inputs with the outputs) with $0$ hidden nodes.

When you add features like $x^3$, this is similar to choosing weights to a few hidden nodes in a single hidden layer. There isn't exactly a $1-1$ correspondence, since to model a function like $x^3$ with sigmoids may take more than one hidden neuron. When you train a neural network, you let it find its own input-to-hidden hidden weights, which has the potential to be better. It may also take more time and it may be inconsistent. You can start with an approximation to logistic regression with extra features, and train the input-to-hidden weights slowly, and this should do better than logistic regression with extra features eventually. Depending on the problem, the training time may be negligible or prohibitive.

One intermediate strategy is to choose a large number of random nodes, similar to what happens when you initialize a neural network, and fix the input-to-hidden weights. The optimization over the *-to-output weights stays linear. This is called an extreme learning machine. It works at least as well as the original logistic regression.

-
"One intermediate strategy is to choose a large number of random nodes, similar to what happens when you initialize a neural network, and fix the input-to-hidden weights. The optimization over the *-to-output weights stays linear." => you mean that there will be a single global maximum for the solution in this case? –  Hugh Perkins Oct 27 '12 at 11:24
For a generic random choice of random hidden nodes, yes. –  Douglas Zare Oct 27 '12 at 11:41