0
$\begingroup$

I have a regression data set with ~1000 features mapping to a single value. Neural networks are consistently 2x more accurate than linear regression, at least with the features I am using.

I am not overfitting either, as I am comparing validation and test errors, and keeping my NNs rather small.

I am wondering how to explain why neural networks are more accurate than linear when using a particular set of features.

Intuitively, we can realize that the NNs are successfully finding nonlinear relationships between the features and the data, and apparently that matters for these particular features.

So here's what I tried:

  1. Calculating $R^2$ values for linear regression. In my case, $R^2=0.98$, suggesting that linear regression is okay.
  2. Calculating PCA eigenvalues of the features, but it looks like they decay rather quickly: PCA of features This suggests that PCA is capable of finding linear combinations of the data successfully?

I am confused by these two results ($R^2=0.98$ and quickly decaying eigenvalues), because it suggests that linear regression should perform well.

Are there other analyses that explain why some features are more suitable for nonlinear regression compared to linear regression?

Maybe a correlation matrix of the features or something?

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ How many observations do you have? How do you measure the accuracy? Did you standardise the features before running the PCA? $\endgroup$
    – Christian Hennig
    Jan 7 at 16:52
  • $\begingroup$ @ChristianHennig About 1 million observations. Accuracy = MAE on validation and test sets. Features are standardized before PCA. $\endgroup$
    – interatomic
    Jan 7 at 17:00
  • 1
    $\begingroup$ PCA has to do with the variance of the features, not the variance in the outcome that can be explained by the features. $\endgroup$
    – Dave
    Jan 7 at 17:20
  • $\begingroup$ You could try plotting the target variable against the ten features constructed by your PCA... this should show any significant nonlinearities, although it won't show interaction effects except incidentally. $\endgroup$
    – jbowman
    Jan 7 at 17:21
  • $\begingroup$ @Dave yeah good point. In this case we should consider variance of the outcome that can be explained by the features, which I thought was $R^2$. $\endgroup$
    – interatomic
    Jan 7 at 17:22

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy