Why a set of highly correlated feature can sometimes perform better than low correlated feature on neural network?

Question

I am trying to solve a classification problem with neural networks. I have extracted large number of feature for each data and trying to do feature selection.

By eliminating features by looking at the correlation matrix and select K best f1 score, the performance is pretty good. However, the performance is not as good when comparing to a subset of my features (same number of feature, same structure of neural network) that is already known to work well with neural networks. That set of feature have some highly correlated features but are removed by the correlation matrix.

What could be the reason causing that?

My guess is that neural networks are capable of understanding non-linear relationships and interactions between features inside the hidden layers. Therefore the feature extracted from corr matrix and select f1 score are good for regression, but cannot find useful feature that are non-linear or have interactions between features. Is that a correct interpretation?

What would be a good way to select features without missing these possibly useful features?

Answer 1

You are correct in that you are likely looking at linear correlations and the neural network is finding non-linear correlations. Your best bet is to let the network do it's job and perform feature learning implicitly. If you insist on reducing dimensionality consider training a noisy auto-encoder prior to your predictive network. This will cast your features in an efficient, lower dimensional space. It's roughly equivalent to kernel PCA.