I have a regression data set with ~1000 features mapping to a single value. Neural networks are consistently 2x more accurate than linear regression, at least with the features I am using.
I am not overfitting either, as I am comparing validation and test errors, and keeping my NNs rather small.
I am wondering how to explain why neural networks are more accurate than linear when using a particular set of features.
Intuitively, we can realize that the NNs are successfully finding nonlinear relationships between the features and the data, and apparently that matters for these particular features.
So here's what I tried:
- Calculating $R^2$ values for linear regression. In my case, $R^2=0.98$, suggesting that linear regression is okay.
- Calculating PCA eigenvalues of the features, but it looks like they decay rather quickly: This suggests that PCA is capable of finding linear combinations of the data successfully?
I am confused by these two results ($R^2=0.98$ and quickly decaying eigenvalues), because it suggests that linear regression should perform well.
Are there other analyses that explain why some features are more suitable for nonlinear regression compared to linear regression?
Maybe a correlation matrix of the features or something?