Neural networks for regression vs. more classical regression methods?

Question

I am interested in learning about when one would use neural networks for a regression problem over a more classical regression method such as least squares.

Is it mostly related to the complexity of the expected relationship between the output and input? Are complicated and strong non-linearity relationships with many dependent variables, and a large amount of data more suitable for neural networks?

Answer 1

Neural nets, (and other nonlinear regression algorithms like trees, Mars, svms, etc.) would be superior (predictive accuracy) when the data is inherently nonlinear.

You can get some nonlinear features in a linear regression by doing things such as manually squaring some terms, etc. , but you are the one deciding the terms.

Nonlinear regression is able to learn the structure / coefficients from your data, ie you aren't having to decide how to inject the nonlinearity in on your own.

Answer 2

There are really no clear answers to this questions in my opinion, but some more practical guidelines I have found to help decide in my own projects:

1. You have a small amount of data for a very difficult problem; neural networks require large amounts of data to work well because in general when we talk about "neural networks" we are not discussing some super shallow network but a model comprised of many layers and units, leading to the need for a large amount of parameters to be estimated. In cases where you have minimal amounts of data I would not be surprised to see classical methods/linear based models outperform neural networks/other non-linear algorithms.
2. You have time constraints or do not have access to the required amount of computing power. Neural networks take much longer to train in general but this is getting better through the use of GPU's and cloud computing. However, compared to say a GLM which is quite trivial to fit on most decent CPU's (and no hyper parameters to tune) there is still a large discrepancy in training times especially if you do not have access to such computing power.
3. You have training data with complex features where feature engineering would be hard, if not, impossible. Take for example a regression problem where you are trying to predict the selling price of cars on your local classifieds based off the photo image taken and also the description given by the seller. These kinds of problems where you have non-tabular data are where neural networks really shine (highest performing) and it's mostly because the training data is not easily interpreted (and therefore, it's hard for us to manually try to find better representations for these types of data = feature engineering).

This does not mean that neural networks are not usable for tabular/structured data, they in general perform quite well given you have the data for it. However, in most cases I've seen other algorithms (mostly forests/gradient boosting/support vector machines) + clever feature engineering out perform neural networks in the cases where feature engineering is possible. Returning to the previous example, say you wish to predict the sales of cars but using characteristics of the car; stuff like color, body type, drive train, weight, gas mileage, MSRP, etc. These are variables that we understand quite well and are easily manipulated without breaking the underlying structure of the data. I'd also include time series forecasting as another area where neural networks seem to be outperformed by classical methods but that's not really relevant to your question.

4. You know for a fact that the relationship between your features and your target is inherently non linear which too be honest, if you have thousands of variables is probably something that would be hard to know.

There are other more practical reasons too. Not many people understand what a neural network is nor do they have the math background. If you are aiming for a model that is easily understood, neural networks are probably not going to sit well with people who perhaps don't trust what they don't understand. In this case, you might just be forced to use a simple linear regression model (I've run into this a lot).

However, if you have the time, why not just try both approaches and through proper validation, see for yourself if the extra complexity + time required to fit neural networks gives enough (if any) improvement to your regression problem over a simple linear one? A linear model will not take very long to fit; start there and then try creating a neural network. If the improvements are substantial/significant, then perhaps your problem is better tackled with a more complex, non-linear model like a neural net.