How to explain why neural networks are more accurate than linear regression?

Question

I have a regression data set with ~1000 features mapping to a single value. Neural networks are consistently 2x more accurate than linear regression, at least with the features I am using.

I am not overfitting either, as I am comparing validation and test errors, and keeping my NNs rather small.

I am wondering how to explain why neural networks are more accurate than linear when using a particular set of features.

Intuitively, we can realize that the NNs are successfully finding nonlinear relationships between the features and the data, and apparently that matters for these particular features.

So here's what I tried:

1. Calculating $R^2$ values for linear regression. In my case, $R^2=0.98$, suggesting that linear regression is okay.
2. Calculating PCA eigenvalues of the features, but it looks like they decay rather quickly: This suggests that PCA is capable of finding linear combinations of the data successfully?

I am confused by these two results ($R^2=0.98$ and quickly decaying eigenvalues), because it suggests that linear regression should perform well.

Are there other analyses that explain why some features are more suitable for nonlinear regression compared to linear regression?

Maybe a correlation matrix of the features or something?

Answer 1

I am wondering how to explain why neural networks are more accurate than linear when using a particular set of features.

Neural networks allow for an enormous number of interactions and nonlinear transformations of the original features. In some regard, neural networks do the feature engineering for you, so you do not have to figure out that a particular interaction matters or that some feature should be squared.

If you get a good fit with a linear model but reliably get an even better fit with a neural network on those same features, it would seem that those nonlinear features and interactions discovered by the neural network matter to the outcome. The math does not work out as cleanly as it does for nested GLMs, but you can think of this as analogous to fitting a model, fitting a more complex model, testing the added features, and getting a low p-value (e.g., partial F-test, "chunk" test in more generality).

To some extent, you are seeing the universal approximation theorems in action. Loosely speaking, the various universal approximation theorems say that a decent$^{\dagger}$ function can be approximated arbitrarily well by a neural network of sufficient size. A linear combination of the raw features has no such guarantee, hence the stronger performance of the neural network.

Where you can get into trouble is that linear models also can involve feature interactions and nonlinear features. The universal approximation theorems say that neural networks can approximate decent functions as well as is desired. The Stone-Weierstrass theorem says about the same about polynomial regressions (which are linear models in the way that statisticians define linear models with linearity in the coefficients instead of in the variables). However, you have to tell the computer what those polynomial features are. You cannot just change model.add(tf.keras.layers.Dense(32, activation='relu')) to model.add(tf.keras.layers.Dense(320, activation='relu')) to get more nonlinearity. This makes it quite easy to increase model flexibility in a neural network compared to a linear model, for better or for worse.

Cheng el. al (2018) have an interesting arXiv paper on polynomial regression vs neural networks.

REFERENCE

Cheng, Xi, et al. "Polynomial regression as an alternative to neural nets." arXiv preprint arXiv:1806.06850 (2018).

$^{\dagger}$This is deliberately vague.