5
$\begingroup$

I'm an academic researcher, working with Convolutional Neural Networks, particularly for image classification. In academic publications, a typical metric for evaluating the performance of a recognition pipeline is the classification accuracy. What I am wondering, is exactly at what point during the training stage this number is taken.

For example, in my experiments, I train the network with back propagation, and reduce the learning rate over time. For this, I observe the testing accuracy, and reduce the rate by a certain amount whenever this accuracy is no longer increasing. However, what I notice is that once the system has converged, and I continue to train with minibatches, the overall testing accuracy still varies after each minibatch by around 1%, even though the average testing error over all minibatches is constant.

So, my questions are:

  1. When reporting the testing accuracy in an academic publication, is it acceptable to simply take the highest accuracy over all these minibatches? Or should something more representative be used, such as an average over all minibatches?

  2. Sometimes, the testing accuracy actually begins to fall as further training is carried out, due to overfitting. Is it acceptable to stop the training at this point, and report this peak testing accuracy, even though the testing dataset is distinct from the training dataset (i.e. the validation set is not a subset of the training dataset)?

$\endgroup$
  • 2
    $\begingroup$ pages.cs.wisc.edu/~shavlik/roc.pdf $\endgroup$ – Reinstate Monica Apr 5 '16 at 13:55
  • $\begingroup$ Thanks, that's an interesting article. In particular though, I'm interested in what the standard approach is for academics in reporting these classification accuracies, particularly for neural networks where training is not deterministic. $\endgroup$ – Karnivaurus Apr 5 '16 at 14:01
  • 4
    $\begingroup$ I have no idea what you're doing. If the system has converged, why are you continuing to train? If it's converged, you're done! Test the model in whatever way is standard in your community and write it up. If you're continuing to run a stochastic simulation after convergence, yeah, it's going to fluctuate because it's stochastic. Metrics of test sets tell you virtually nothing about the out-of-sample performance of the model, which is the only relevant performance information. $\endgroup$ – Reinstate Monica Apr 5 '16 at 14:05
  • 3
    $\begingroup$ If the training is done ("converged"), and you're continuing to poke around for a random batch that makes your results better, it sounds a lot like you're on a fishing expedition for the samples that will make your results look artificially better. But your third sentence sounds like a contradiction: you're not training to improve results, but you are training to improve test accuracy? What does that mean? $\endgroup$ – Reinstate Monica Apr 5 '16 at 14:54
  • 1
    $\begingroup$ I'd like to say that it is not academic practice to be deceptive in publications, but recent examples of academic misconduct demonstrate that is not strictly the case that authors are aboveboard in presentation of their findings. But I'm probably naive for insisting that people be honest. $\endgroup$ – Reinstate Monica Apr 5 '16 at 15:22
3
$\begingroup$

Before I answer your question let me say, in general it is considered good practice to have a Validation set that is completely distinct from your Test set. You may or may not be aware of this fact, but you seem to gloss over it in your question, so I wanted to make that explicitly clear.

The answer to your question would be an average across all mini-batches of your Test set, as the Test set is supposed to represent an unbiased representation of how that NN may perform in the wild. Another way of stating this would be that the test set (since you haven't tuned your hyper-parameters to do well on that set) should represent how well your network generalizes, which is the goal for any network.

It would likely be considered deceptive academic practice to cherry-pick the best mini-batch for publishing results, and generally any published results should be able to be reproduced by other researchers. If you artificially conflate your accuracy by choosing the best mini-batch, that would make replication of your results difficult for other researchers and would likely lead other researchers to question the validity of your claims.

Examples of this can be seen in other areas of Science: many researchers have swiftly ruined their careers by publishing results that are not able to be reproduced by other researchers.

While I can't explicitly tell you whether or not this is typical of academic publications in this field of study, it is definitely unethical, probably immoral, and, as you have already stated yourself, not good practice.

It is likely that as this field grows in maturity, more attention will be paid to replication of results, and in doing so, will make the ability to replicate results vital to not only the integrity of the researcher, but also whether or not the research is generally accepted.

Please note that reproducability is a standard in all Scientific endeavors, and Computer Scientists are not the only ones facing this issue. For instance, I just ran across this BBC article about reproducability. This article (tangentially) addresses some of the points you have brought up with this question. This article also addresses the issue of, and need for, reproducability in general.

$\endgroup$
  • $\begingroup$ Absolutely. These are very important points, +1 for making them explicit $\endgroup$ – user20160 Feb 26 '18 at 10:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.