I'm training a classifier with a set of training data and checking the result with a set of test data. I'm using
sklearn score based on all data and iteratively generating a random number of layers and neurons per layer to see what kind of score I get.
But, most results I get yield a score that is very close to 1, that is, for a variable number of layers and neurons, the MLP basically just almost always gets it right. "Almost" being an important word, since I'm yet to get a perfect result on the test data.
I'm wondering if there is a method or rule of thumb to pick one between several options of layouts. Maybe the less neurons the better? But that would work to compare options yielding perfect score. Maybe higher score per neuron times number of layers? But that would heavily penalize a high number of neurons over an accurate result. Looking at the scatter plot below, I'm wondering if I should pick the lucky outlier with the best accuracy of the test data, or if i should compromise and get some other point in the pareto front.
I've worked in the past with Akaike's Criterion is there something similar for MLP? Would it be possible to compute the likelihood of the MLP given the data sets I have?