0
$\begingroup$

I'm training a classifier with a set of training data and checking the result with a set of test data. I'm using sklearn score based on all data and iteratively generating a random number of layers and neurons per layer to see what kind of score I get.

But, most results I get yield a score that is very close to 1, that is, for a variable number of layers and neurons, the MLP basically just almost always gets it right. "Almost" being an important word, since I'm yet to get a perfect result on the test data.

I'm wondering if there is a method or rule of thumb to pick one between several options of layouts. Maybe the less neurons the better? But that would work to compare options yielding perfect score. Maybe higher score per neuron times number of layers? But that would heavily penalize a high number of neurons over an accurate result. Looking at the scatter plot below, I'm wondering if I should pick the lucky outlier with the best accuracy of the test data, or if i should compromise and get some other point in the pareto front.

I've worked in the past with Akaike's Criterion is there something similar for MLP? Would it be possible to compute the likelihood of the MLP given the data sets I have?

enter image description here

$\endgroup$
  • $\begingroup$ This may be helpful for AIC estimation stats.stackexchange.com/a/174568/289381 $\endgroup$ – user289381 Jul 28 '20 at 9:11
  • $\begingroup$ sklearn has different metrics defined - I assume that you refer to the accuracy $\endgroup$ – Match Maker EE Jul 28 '20 at 12:17
  • $\begingroup$ @ping : I understand how to count parameters in an MLP, but not how I would compute the likelihood. $\endgroup$ – Mefitico Jul 28 '20 at 16:32
  • $\begingroup$ @MatchMakerEE I'm referring to the score method of the class sklearn.neural_network.MLPClassifier ref doc $\endgroup$ – Mefitico Jul 28 '20 at 16:34
1
$\begingroup$

Model selection for neural networks is complex. You can vary their 'width', the number of hidden nodes in one layer, and their 'depth', the number of hidden layers.

Specifically for image processing, deep convolutional neural networks have proven really successful. They explore the deep structure in images in a bottom-up manner. You do not mention that your NNs are trained with image data.

Occam's razor: use the simplest possible model that can predict well ("entities should not be multiplied without necessity"). This means that you should choose the MLP with the smallest number of parameters that is well-performing. It is the total number of parameters (weights + bias terms) that needs to be kept minimal. Note that more restricted MLPs (with fewer parameters) are often more difficult to train.

I always split the training set into a training adjustment set and a training generalization set (The separate test set is kept out of the loop, until all training and model selection has taken place). The split into two training subsets is performed by a random generator. Your training algorithm fits the MLP-parameters on the training adjustment set, using backpropagation, or some if its later variants. You can now use cross validation on the training generalization set to compare the generalization ability of different MLP topologies. You can plot the number of parameters for each MLP topology versus the accuracy as computed on the training generalization set.

Model selection means choosing the MLP that yields the best trade-off between the number of parameters and the accuracy as computed on the training generalization set. Finally, you evaluate your best MLP using the independent test set.

$\endgroup$
  • $\begingroup$ From what I've understood about your answer (which is better written than I would have) I've basically done what you've said. But afterwards I'm asking "Now what?". I'm listing the scores I get for different depths and widths, and I'm almost always getting a score of 1 for my "training adjustment set". For my "training generalization set", I get one score of 0.977250 with 4 hidden layers and a few of 0.975272 scores for 2 or 3 layers (and worse results using up to 6 layers and several combinations of widths). I'm still running other combinations. This is where I'm getting. $\endgroup$ – Mefitico Jul 28 '20 at 16:40
  • $\begingroup$ Well, really good performance. That is, if about that high accuracy is also achieved on an independent test set. With so many hidden layers and therefore parameters, you can compare your MLP with the performance of a linear and quadratic discriminant, perhaps a random forest classifier. These much simpler models give a baseline for your performance. What ever your result will be on these much simpler models, you get insight into the added performance by your complex MLP-classifiers. My final advice is the choose the MLP with the fewest number of parameters.. $\endgroup$ – Match Maker EE Jul 28 '20 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.