I have a regression data set with ~1000 features mapping to a single value. Neural networks are consistently 2x more accurate than linear regression, at least with the features I am using.

I am not overfitting either, as I am comparing validation and test errors, and keeping my NNs rather small.

I am wondering how to explain why neural networks are more accurate than linear when using a particular set of features.

Intuitively, we can realize that the NNs are successfully finding nonlinear relationships between the features and the data, and apparently that matters for these particular features.

So here's what I tried:

  1. Calculating $R^2$ values for linear regression. In my case, $R^2=0.98$, suggesting that linear regression is okay.
  2. Calculating PCA eigenvalues of the features, but it looks like they decay rather quickly: PCA of features This suggests that PCA is capable of finding linear combinations of the data successfully?

I am confused by these two results ($R^2=0.98$ and quickly decaying eigenvalues), because it suggests that linear regression should perform well.

Are there other analyses that explain why some features are more suitable for nonlinear regression compared to linear regression?

Maybe a correlation matrix of the features or something?

  • 2
    $\begingroup$ How many observations do you have? How do you measure the accuracy? Did you standardise the features before running the PCA? $\endgroup$ Commented Jan 7, 2023 at 16:52
  • $\begingroup$ @ChristianHennig About 1 million observations. Accuracy = MAE on validation and test sets. Features are standardized before PCA. $\endgroup$ Commented Jan 7, 2023 at 17:00
  • 2
    $\begingroup$ PCA has to do with the variance of the features, not the variance in the outcome that can be explained by the features. $\endgroup$
    – Dave
    Commented Jan 7, 2023 at 17:20
  • $\begingroup$ You could try plotting the target variable against the ten features constructed by your PCA... this should show any significant nonlinearities, although it won't show interaction effects except incidentally. $\endgroup$
    – jbowman
    Commented Jan 7, 2023 at 17:21
  • 2
    $\begingroup$ That PCA can reduce the dimensionality of the data with little loss of information is no indication whatsoever that any of the observed features are at all related to the value you're trying to predict. If I give you a dataset that contains individuals' heights measured in inches, centimeters, feet, furlongs, and fathoms, PCA will reduce that to a single dimension with no loss of variability, but that of course doesn't mean the features are useful for an arbitrary classification task like predicting the person's social security number. $\endgroup$ Commented Apr 18, 2023 at 17:44

1 Answer 1


I am wondering how to explain why neural networks are more accurate than linear when using a particular set of features.

Neural networks allow for an enormous number of interactions and nonlinear transformations of the original features. In some regard, neural networks do the feature engineering for you, so you do not have to figure out that a particular interaction matters or that some feature should be squared.

If you get a good fit with a linear model but reliably get an even better fit with a neural network on those same features, it would seem that those nonlinear features and interactions discovered by the neural network matter to the outcome. The math does not work out as cleanly as it does for nested GLMs, but you can think of this as analogous to fitting a model, fitting a more complex model, testing the added features, and getting a low p-value (e.g., partial F-test, "chunk" test in more generality).

To some extent, you are seeing the universal approximation theorems in action. Loosely speaking, the various universal approximation theorems say that a decent$^{\dagger}$ function can be approximated arbitrarily well by a neural network of sufficient size. A linear combination of the raw features has no such guarantee, hence the stronger performance of the neural network.

Where you can get into trouble is that linear models also can involve feature interactions and nonlinear features. The universal approximation theorems say that neural networks can approximate decent functions as well as is desired. The Stone-Weierstrass theorem says about the same about polynomial regressions (which are linear models in the way that statisticians define linear models with linearity in the coefficients instead of in the variables). However, you have to tell the computer what those polynomial features are. You cannot just change model.add(tf.keras.layers.Dense(32, activation='relu')) to model.add(tf.keras.layers.Dense(320, activation='relu')) to get more nonlinearity. This makes it quite easy to increase model flexibility in a neural network compared to a linear model, for better or for worse.

Cheng el. al (2018) have an interesting arXiv paper on polynomial regression vs neural networks.


Cheng, Xi, et al. "Polynomial regression as an alternative to neural nets." arXiv preprint arXiv:1806.06850 (2018).

$^{\dagger}$This is deliberately vague.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.