It seems that it is possible to get similar results to a neural network with a multivariate linear regression in some cases, and multivariate linear regression is super fast and easy.

Under what circumstances can neural networks give better results than multivariate linear regression?

up vote 27 down vote accepted

Neural networks can in principle model nonlinearities automatically (see the universal approximation theorem), which you would need to explicitly model using transformations (splines etc.) in linear regression.

The caveat: the temptation to overfit can be (even) stronger in neural networks than in regression, since adding hidden layers or neurons looks harmless. So be extra careful to look at out-of-sample prediction performance.

  • Ok. I guess a question in my mind is, to what extent can I replicate similar behavior by augmenting my input data with quadratic and cubic terms? – Hugh Perkins Oct 27 '12 at 9:32
  • 3
    Actually, you can probably approximate NNs with appropriately transformed regressors in a linear regression as closely as you want (and vice versa). Better practice than quadratics and cubics are splines, though - I heartily recommend Harrell's textbook "Regression Modeling Strategies". – Stephan Kolassa Oct 27 '12 at 10:42
  • Ok. Is it reasonable to assume that training time will be faster for linear regression on transformed data, or will the training times be approximately similar? Will the solution for the linear regression on transformed data have a single global maximum, or will it have lots of local minimum as for neural networks? (Edit: I guess no matter how the inputs are transformed, the solution to the linear regression is just the pseudoinverse of the design matrix multiplied by something-something and therefore is always either unique or singular?) – Hugh Perkins Oct 27 '12 at 11:22
  • 2
    Training times will of course depend on input dimensions (few/many observations, few/many predictors). Linear regression involves a single (pseudo-)inverse (yes, uniqueness/singularity even with transformed regressors holds), whereas NNs are typically trained in an iterative way, but iterations don't involve matrix inversions, so each iteration is faster - you typically stop the training based on some criterion designed to stop you from overfitting. – Stephan Kolassa Oct 27 '12 at 11:35
  • 1
    @Yamcha: my understanding of the universal approximation theorem is that the dimensionality in principle does not matter. (Of course, this is an asymptotic result. I'd expect that you'd need horrendous amounts of data for the NN to be better than a fine-tuned polynomial regression. Starts to sound like Deep Learning...) – Stephan Kolassa Oct 28 '16 at 6:06

You mention linear regression. This is related to logistic regression, which has a similar fast optimization algorithm. If you have bounds on the target values, such as with a classification problem, you can view logistic regression as a generalization of linear regression.

Neural networks are strictly more general than logistic regression on the original inputs, since that corresponds to a skip-layer network (with connections directly connecting the inputs with the outputs) with $0$ hidden nodes.

When you add features like $x^3$, this is similar to choosing weights to a few hidden nodes in a single hidden layer. There isn't exactly a $1-1$ correspondence, since to model a function like $x^3$ with sigmoids may take more than one hidden neuron. When you train a neural network, you let it find its own input-to-hidden hidden weights, which has the potential to be better. It may also take more time and it may be inconsistent. You can start with an approximation to logistic regression with extra features, and train the input-to-hidden weights slowly, and this should do better than logistic regression with extra features eventually. Depending on the problem, the training time may be negligible or prohibitive.

One intermediate strategy is to choose a large number of random nodes, similar to what happens when you initialize a neural network, and fix the input-to-hidden weights. The optimization over the *-to-output weights stays linear. This is called an extreme learning machine. It works at least as well as the original logistic regression.

  • 1
    "One intermediate strategy is to choose a large number of random nodes, similar to what happens when you initialize a neural network, and fix the input-to-hidden weights. The optimization over the *-to-output weights stays linear." => you mean that there will be a single global maximum for the solution in this case? – Hugh Perkins Oct 27 '12 at 11:24
  • 1
    For a generic random choice of random hidden nodes, yes. – Douglas Zare Oct 27 '12 at 11:41
  • 1
    great post - providing context for [LR, LogR, NN, ELM]. Your comment about the LogR being a skip-layer NN seems obvious after being pointed out but is a nice insight. – javadba Dec 21 '15 at 14:02

Linear Regression aims to separate the data that is linearly separable, yes you may use additional third> degree polynomials but in that way you indicated again some assumptions about the data you have since you define the objective function's structure. In Neural Net. generally you have input layer that creates the linear separators for the data you have and hidden layer ANDs the regions that bounds some classes and last layer ORs all these regions. In that way all the data you have is able to be classified with non linear way, also all these process is going with internally learned weights and defined functions. In addition increasing the feature number for Linear Regression is opposed to "Curse of dimensionality". In addition some applications need more probabilistic results than constant numbers as output. Thus a NN with logistic function will be more suitable for such purposes (Of course there is also logistic regression suffers form the facts I told).

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.