2
$\begingroup$

Histogram

Let's say I have the histogram above which reports performances of some neural networks. The y axis is the bin size, while the x axis is the error, so low errors = high performance.

Out of these 15 networks, how would I go about selecting the best one? Looking at the mean or the median error comes to mind, but I can think of many cases where mean and the median could be misleading. Is there a way to pick the consistently high performing network?

Here it is obvious that MLP 10 has the best performance, but sometimes evaluating a network becomes much harder.

The data is not necessarily normally distributed.

$\endgroup$
0
$\begingroup$

In terms of evaluating the performance of your neural network, you typically want to use some kind of estimate for a generalisation error, like hold-out validation (testing on a separate dataset) or K-fold cross-validation.

Assuming you use a separate dataset for comparison, the reasonable thing to do is to use the loss function you have used for training your neural network as an evaluation metric.

The most common choices for loss functions are the mean squared error and cross-entropy.

Mean squared error is a typical choice for regression problems:

$$ MSE = \frac{1}{N} \sum_{i=1}^N (Y_i - \hat{Y}_i )^2 $$

where $Y_i$ is the true regression value and $\hat{Y_i}$ is your neural network's prediction.

While the cross-entropy is used for classification problems:

$$ C = - \sum_{i=1}^N y_{o,c} \log p_{o,c} $$

where $y_{o,c} $ is a binary indicator of correct classification, and $p_{o,c}$ is the predicted probability for that class.

If you used the above two as an error metric, I would take the average of the errors on each histogram and choose the minimum.

Otherwise, I would calculate the value the loss functions on your separate dataset, and choose the neural network which gives the minimal error.

$\endgroup$
  • $\begingroup$ Thank you. In this case, each network controls a robot performing a task, and the error is reported as above. If I sum up all the squared errors, this will give me a ranking of the networks, which is what I need. This will also inflate the large errors though. What if I think errors around 0-250 are acceptable, and an error of 2000 and an error of 10000 are both unacceptable, so there shouldn't be much difference in large errors. Does this mean the error I used for training is not quite good? Is there an error function with diminishing returns on high errors? $\endgroup$ – ovunctuzel Feb 6 '18 at 23:36
  • $\begingroup$ I think the penalising more heavily won't solve the large errors, in my experience, what typically works is taking a look at the erroneous examples and considering some kind of extension of the training set based on it. As far as I'm concerned there is no loss function with diminishing returns behaviour, but every week somebody finds out a new kind of loss function or metric or finds out some kind of heuristic for a previously intractable loss function. Try a few alternative loss functions and see if they change performance. $\endgroup$ – boomkin Feb 7 '18 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.