How to evaluate a histogram? 
Let's say I have the histogram above which reports performances of some neural networks. The y axis is the bin size, while the x axis is the error, so low errors = high performance. 
Out of these 15 networks, how would I go about selecting the best one? Looking at the mean or the median error comes to mind, but I can think of many cases where mean and the median could be misleading. Is there a way to pick the consistently high performing network?
Here it is obvious that MLP 10 has the best performance, but sometimes evaluating a network becomes much harder.
The data is not necessarily normally distributed. 
 A: In terms of evaluating the performance of your neural network, you typically want to use some kind of estimate for a generalisation error, like hold-out validation (testing on a separate dataset) or K-fold cross-validation.
Assuming you use a separate dataset for comparison, the reasonable thing to do is to use the loss function you have used for training your neural network as an evaluation metric. 
The most common choices for loss functions are the mean squared error and cross-entropy. 
Mean squared error is a typical choice for regression problems:
$$ MSE = \frac{1}{N} \sum_{i=1}^N (Y_i - \hat{Y}_i )^2  $$
where $Y_i$ is the true regression value and $\hat{Y_i}$ is your neural network's prediction.
While the cross-entropy is used for classification problems:
$$ C = - \sum_{i=1}^N y_{o,c} \log p_{o,c} $$
where $y_{o,c} $ is a binary indicator of correct classification, and $p_{o,c}$ is the predicted probability for that class.
If you used the above two as an error metric, I would take the average of the errors on each histogram and choose the minimum.
Otherwise, I would calculate the value the loss functions on your separate dataset, and choose the neural network which gives the minimal error.
