Criterion to Pick Number of hidden layers for MLP based on score

Question

I'm training a classifier with a set of training data and checking the result with a set of test data. I'm using sklearn score based on all data and iteratively generating a random number of layers and neurons per layer to see what kind of score I get.

But, most results I get yield a score that is very close to 1, that is, for a variable number of layers and neurons, the MLP basically just almost always gets it right. "Almost" being an important word, since I'm yet to get a perfect result on the test data.

I'm wondering if there is a method or rule of thumb to pick one between several options of layouts. Maybe the less neurons the better? But that would work to compare options yielding perfect score. Maybe higher score per neuron times number of layers? But that would heavily penalize a high number of neurons over an accurate result. Looking at the scatter plot below, I'm wondering if I should pick the lucky outlier with the best accuracy of the test data, or if i should compromise and get some other point in the pareto front.

I've worked in the past with Akaike's Criterion is there something similar for MLP? Would it be possible to compute the likelihood of the MLP given the data sets I have?

Answer 1

Model selection for neural networks is complex. You can vary their 'width', the number of hidden nodes in one layer, and their 'depth', the number of hidden layers.

Specifically for image processing, deep convolutional neural networks have proven really successful. They explore the deep structure in images in a bottom-up manner. You do not mention that your NNs are trained with image data.

Occam's razor: use the simplest possible model that can predict well ("entities should not be multiplied without necessity"). This means that you should choose the MLP with the smallest number of parameters that is well-performing. It is the total number of parameters (weights + bias terms) that needs to be kept minimal. Note that more restricted MLPs (with fewer parameters) are often more difficult to train.

I always split the training set into a training adjustment set and a training generalization set (The separate test set is kept out of the loop, until all training and model selection has taken place). The split into two training subsets is performed by a random generator. Your training algorithm fits the MLP-parameters on the training adjustment set, using backpropagation, or some if its later variants. You can now use cross validation on the training generalization set to compare the generalization ability of different MLP topologies. You can plot the number of parameters for each MLP topology versus the accuracy as computed on the training generalization set.

Model selection means choosing the MLP that yields the best trade-off between the number of parameters and the accuracy as computed on the training generalization set. Finally, you evaluate your best MLP using the independent test set.