I am using a neural network with one layer of 20 hidden units (using the package nnet) for a classification problem where I have around 12 possible outcomes. I have around 4000 observations, and I am using 10 fold cross-validation to evaluate the network.
Now, I am not particularly interested in the classification task perse, bu in testing a hypothesis about the predictors. I have two sets of predictors, say A and B, and if my theory is right then set A should play a role in the classification task, but set B should not play a role in the classification task.
Now, what I was expecting to see was that using set A I would get a given classification accuracy, and that adding set B I would get more or less the same classification accuracy. However, what I see is that the network becomes noticeably worse. It goes from an accuracy of around 86%, to something like 66%.
I tried increasing the number of iterations to 2000 until each training started to always converge, but that did not change anything. I am of course happy with the result, but I do not understand how adding predictors can make a neural network perform worse. What could be going on here?