Criterion to Pick Number of hidden layers for MLP based on score

I'm training a classifier with a set of training data and checking the result with a set of test data. I'm using sklearn score based on all data and iteratively generating a random number of layers and neurons per layer to see what kind of score I get.

But, most results I get yield a score that is very close to 1, that is, for a variable number of layers and neurons, the MLP basically just almost always gets it right. "Almost" being an important word, since I'm yet to get a perfect result on the test data.

I'm wondering if there is a method or rule of thumb to pick one between several options of layouts. Maybe the less neurons the better? But that would work to compare options yielding perfect score. Maybe higher score per neuron times number of layers? But that would heavily penalize a high number of neurons over an accurate result. Looking at the scatter plot below, I'm wondering if I should pick the lucky outlier with the best accuracy of the test data, or if i should compromise and get some other point in the pareto front.

I've worked in the past with Akaike's Criterion is there something similar for MLP? Would it be possible to compute the likelihood of the MLP given the data sets I have?

• This may be helpful for AIC estimation stats.stackexchange.com/a/174568/289381 – user289381 Jul 28 '20 at 9:11
• sklearn has different metrics defined - I assume that you refer to the accuracy – Match Maker EE Jul 28 '20 at 12:17
• @ping : I understand how to count parameters in an MLP, but not how I would compute the likelihood. – Mefitico Jul 28 '20 at 16:32
• @MatchMakerEE I'm referring to the score method of the class sklearn.neural_network.MLPClassifier ref doc – Mefitico Jul 28 '20 at 16:34