Neural network becomes worse with more predictors

Question

I am using a neural network with one layer of 20 hidden units (using the package nnet) for a classification problem where I have around 12 possible outcomes. I have around 4000 observations, and I am using 10 fold cross-validation to evaluate the network.

Now, I am not particularly interested in the classification task perse, bu in testing a hypothesis about the predictors. I have two sets of predictors, say A and B, and if my theory is right then set A should play a role in the classification task, but set B should not play a role in the classification task.

Now, what I was expecting to see was that using set A I would get a given classification accuracy, and that adding set B I would get more or less the same classification accuracy. However, what I see is that the network becomes noticeably worse. It goes from an accuracy of around 86%, to something like 66%.

I tried increasing the number of iterations to 2000 until each training started to always converge, but that did not change anything. I am of course happy with the result, but I do not understand how adding predictors can make a neural network perform worse. What could be going on here?

Answer 1

This outcome would be expected. Adding inputs to a neural network that don't help predict the output should give the network a hard time. Think of it as drowning out the inputs that actually do help with the classification task.

One of the reasons max pooling helps convolutional neural networks is because it gets rid of information that it thinks is less useful and concentrates more on important things.

Another reason is because noisy data is much more prone to overfitting. Adding useless inputs is the exact same thing as just having noisier data.

Answer 2

Now, I am not particularly interested in the classification task per se, bu in testing a hypothesis about the predictors. I have two sets of predictors, say A and B, and if my theory is right then set A should play a role in the classification task, but set B should not play a role in the classification task.

What exactly would it prove? Imagine that instead of using only the neural network, you used neural network and XGBoost, and one of the algorithm would work better on set A, while the other, on set B, what would that prove?

That something "should not play a role in the classification task" is not a valid hypothesis to test. Trying to answer this would give you the wrong answer, to the wrong question.

Moreover, machine learning algorithms are in many cases sensitive to the choice of hyperparameters. How would you tune the hyperparameters? Would you tune them in terms of model performance on set A, or on set B? This would give you biased result. You could choose to tune based on overall performance, but how would you measure it? If you would sum the error metrics from A and B, then this does not prevent one of the sets to having greater influence then the other (e.g. better parameter choice of the set with worse initial error would lead to greater improvement in performance, so will have greater weight on overall score).