7
$\begingroup$

I'm interested in understanding which neural network architecture is currently the state of the art (sometimes abbreviated "SOTA") with respect to standard image classification tasks such as MNIST, STLN-10 and CIFAR. This is challenging because new results are published frequently, and it can be hard to keep up. Is there a resource or website which tracks the best results for these tasks?

$\endgroup$
  • 1
    $\begingroup$ what about choosing one of the three answers, so that the question doesn't remain open? Or, in case you don't choose yours, specify what could be modified about the other two in order to make them acceptable? $\endgroup$ – DeltaIV Sep 3 '18 at 8:42
  • 1
    $\begingroup$ @SycoraxsaysReinstateMonica Added! $\endgroup$ – HelloGoodbye Jan 7 at 9:46
7
$\begingroup$

A "leaderboard" of sorts is maintained at this website, "Classification datasets results". The maintainers attempt to keep track of published results of various neural network architectures.

The leaderboard is not solely restricted to CNNs per se -- any network is admissible. But because all of the tasks tracked on the leaderboard are image tasks (as of this writing), it's likely that many of the networks will be CNNs since they are very effective at image tasks.

$\endgroup$
6
$\begingroup$

The best suggestion is from shimao:

typically any new paper which claims good or state-of-the-art performance on any task will have a fairly comprehensive results table comparing with previous results, which can be good way to keep track.

Any leaderboard will soon become useless, because they're basically always maintained by (undergrad/grad) students, who stop updating them as soon as they get their degree/land a job. Anyway, if CIFAR-10 and CIFAR-100 are good enough for you, this is pretty good:

https://github.com/arunpatala/cifarSOTA

This one is more general (includes ImageNet) and it has more recent results:

https://github.com/Lextal/SotA-CV

This is the one I used to use, but the owner has stopped updating it, as it often happens:

https://github.com/RedditSota/state-of-the-art-result-for-machine-learning-problems/

Finally, you may be interested in this Jupyter notebook released just today by Ali Rahimi, based on data scraped from LSVRC and the COCO website.

One last note: if you’re looking for the latest results because you want to compare your results to SotA, great. However, if your goal is applying the “best” architecture on ImageNet to an industrial application using transfer learning, you should know (if you don’t already) that the latest architectures are worse, in terms of translation invariance, than the older ones. This is a risk if your dataset doesn’t have photographer bias of if you don’t have enough compute and data to retrain the architecture on a more useful image distribution. See the excellent preprint:

Azulay & Weiss, 2018, more “Why do deep convolutional networks generalize so poorly to small image transformations?”

$\endgroup$
5
$\begingroup$

DenseNet is a generic successor to ResNet and achieves 3.46% error on CIFAR-10 and 17.18 on C-100. Compare to 3.47 and 24.28 mentioned on the leaderboard.

Shake-shake, Shake-drop and possibly other variants are regularization techniques which can be used any ResNet-like architectures, and achieves 2.86/2.31% error on C-10 and 15.85/12.19 on C-100 (shake-shake/shake-drop). These techniques work on only multi-branch architectures, which is why I mention them even though they are not strictly architectures in themselves.

Efficient Neural Architecture Search (using reinforcement learning to search for architectures) finds a network which achieves 2.89% error on C-10, using the Cutout regularization technique. Performance is 3.54% without cutout.

In summary: Dense Net and possibly some ENAS produced network may perform slightly better than ResNet, but the use of sophisticated regularization techniques makes the comparison admittedly difficult.

I don't know of any leaderboard which is really up to date, but typically any new paper which claims good or state-of-the-art performance on any task will have a fairly comprehensive results table comparing with previous results, which can be good way to keep track.

$\endgroup$
2
$\begingroup$

For checking state-of-the-art neural network architectures (and other machine learning models) in various application domains, there is now a page called paperswithcode.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.