Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
It seems that it is possible to get similar results to a neural network with a multivariate linear regression in some cases, and multivariate linear regression is super fast and easy.
Under what circumstances can neural networks give better results than multivariate linear regression?
Neural networks can in principle model nonlinearities automatically (see the universal approximation theorem), which you would need to explicitly model using transformations (splines etc.) in linear regression.
The caveat: the temptation to overfit can be (even) stronger in neural networks than in regression, since adding hidden layers or neurons looks harmless. So be extra careful to look at out-of-sample prediction performance.
You mention linear regression. This is related to logistic regression, which has a similar fast optimization algorithm. If you have bounds on the target values, such as with a classification problem, you can view logistic regression as a generalization of linear regression.
Neural networks are strictly more general than logistic regression on the original inputs, since that corresponds to a skip-layer network (with connections directly connecting the inputs with the outputs) with $0$ hidden nodes.
When you add features like $x^3$, this is similar to choosing weights to a few hidden nodes in a single hidden layer. There isn't exactly a $1-1$ correspondence, since to model a function like $x^3$ with sigmoids may take more than one hidden neuron. When you train a neural network, you let it find its own input-to-hidden hidden weights, which has the potential to be better. It may also take more time and it may be inconsistent. You can start with an approximation to logistic regression with extra features, and train the input-to-hidden weights slowly, and this should do better than logistic regression with extra features eventually. Depending on the problem, the training time may be negligible or prohibitive.
One intermediate strategy is to choose a large number of random nodes, similar to what happens when you initialize a neural network, and fix the input-to-hidden weights. The optimization over the *-to-output weights stays linear. This is called an extreme learning machine. It works at least as well as the original logistic regression.
Linear Regression aims to separate the data that is linearly separable, yes you may use additional third> degree polynomials but in that way you indicated again some assumptions about the data you have since you define the objective function's structure. In Neural Net. generally you have input layer that creates the linear separators for the data you have and hidden layer ANDs the regions that bounds some classes and last layer ORs all these regions. In that way all the data you have is able to be classified with non linear way, also all these process is going with internally learned weights and defined functions. In addition increasing the feature number for Linear Regression is opposed to "Curse of dimensionality". In addition some applications need more probabilistic results than constant numbers as output. Thus a NN with logistic function will be more suitable for such purposes (Of course there is also logistic regression suffers form the facts I told).
