1
$\begingroup$

It is common to depict the error rates of types of neural networks in a table, for example, see the MNIST website.

However, because of the non-determinism caused by weight initialization the actual error rates may vary even under a single setting of hyperparameters.

My question is therefore: what statistic over multiple runs is usually depicted? Are the reported error rates the mean error rates? Are they the minimum error rates?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

As you can see in the website, samples are splitted in two files: a training file and a test file. Accuracy is given on the test set, so that results are comparable.

Now, in practice one usually employs k-fold cross validation in order to average away those effects you mention. The average value over the folds is reported.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Since a separate testing set is given, it is unlikely that cross-validation has been used. Moreover, the nondeterminism causes different runs to perform wildly different. It would therefore not be such a good idea average over the performance of these networks. I do not think they have used cross validation to calculate the scores depicted. $\endgroup$
    – Angelorf
    Commented Aug 12, 2014 at 12:38
  • $\begingroup$ That's exactly my point. The point is that you need to ensure that everybody give their accuracy estimates on the same test data. So, no, tt has not been used, but what I mean is that is often done that way. I said that in relation to your last questions. Maybe I should have been more explicit, sorry. $\endgroup$
    – jpmuc
    Commented Aug 12, 2014 at 13:14
  • $\begingroup$ I do not see how you have answered any of my questions. I know that accuracy on the test set is calculated, but how do the accuracies of multiple runs end up in a single statistic? $\endgroup$
    – Angelorf
    Commented Aug 12, 2014 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.