When does larger network decrease performance?

Question

I am training a deep neural network of mostly one-hot encoded features, which are very sparse (500 columns, mostly zeros).

My network architecture of Dense 64 (w/ dropout)->128 (w/ dropout)->128 (w/ dropout)->256 (w/ dropout) performs significantly worse than a single layer Dense 128...why might this be? I doubt that the deeper network is getting caught in a local minimum, because the first few batches of the simpler network perform better than the 20th epoch of the deeper net. All activations are ReLU

Answer 1

Generally speaking, your network should perform better as you increase the number of hidden layers since the network would learn and identify specific features that can directly impact your output.

Assuming that your output won't be more than 128, generally, the rule of thumb is that the number of neurons in a hidden layer is between 1 to the number of inputs. Maybe you should try that out. I've also heard of some saying that another metric is that the number of neurons are between 1 and the number of inputs minus the number of outputs

Answer 2

You didn't mention whether your training error is bad or whether it is the test error.

Generally, bigger networks should allow you to achieve a very small training error - however, if the capacity of your model is way bigger than what is needed for the problem. you will start overfitting on your training data. This would result in a very high error on the test set which would explain why you get better results with a small model.

For more details on this check out chapter 5 of the deeplearning book: http://www.deeplearningbook.org/contents/ml.html