I am trying to solve a classification problem with neural networks. I have extracted large number of feature for each data and trying to do feature selection.

By eliminating features by looking at the correlation matrix and select K best f1 score, the performance is pretty good. However, the performance is not as good when comparing to a subset of my features (same number of feature, same structure of neural network) that is already known to work well with neural networks. That set of feature have some highly correlated features but are removed by the correlation matrix.

What could be the reason causing that?

My guess is that neural networks are capable of understanding non-linear relationships and interactions between features inside the hidden layers. Therefore the feature extracted from corr matrix and select f1 score are good for regression, but cannot find useful feature that are non-linear or have interactions between features. Is that a correct interpretation?

What would be a good way to select features without missing these possibly useful features?


1 Answer 1


You are correct in that you are likely looking at linear correlations and the neural network is finding non-linear correlations. Your best bet is to let the network do it's job and perform feature learning implicitly. If you insist on reducing dimensionality consider training a noisy auto-encoder prior to your predictive network. This will cast your features in an efficient, lower dimensional space. It's roughly equivalent to kernel PCA.

  • $\begingroup$ Thanks for the reply. I tried to incorporate the entire feature set as input but that didn't improve the performance. Would a wider or deeper nn helps? $\endgroup$
    – Jack
    Apr 9, 2020 at 9:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.