I'm interested in understanding which neural network architecture is currently the state of the art (sometimes abbreviated "SOTA") with respect to standard image classification tasks such as MNIST, STLN-10 and CIFAR. This is challenging because new results are published frequently, and it can be hard to keep up. Is there a resource or website which tracks the best results for these tasks?
$\begingroup$
$\endgroup$
0
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
The best suggestion is from shimao:
typically any new paper which claims good or state-of-the-art performance on any task will have a fairly comprehensive results table comparing with previous results, which can be good way to keep track.
Any leaderboard will soon become useless, because they're basically always maintained by (undergrad/grad) students, who stop updating them as soon as they get their degree/land a job. Anyway, if CIFAR-10 and CIFAR-100 are good enough for you, this is pretty good:
https://github.com/arunpatala/cifarSOTA
This one is more general (includes ImageNet) and it has more recent results:
https://github.com/Lextal/SotA-CV
This is the one I used to use, but the owner has stopped updating it, as it often happens:
https://github.com/RedditSota/state-of-the-art-result-for-machine-learning-problems/
Finally, you may be interested in this Jupyter notebook released just today by Ali Rahimi, based on data scraped from LSVRC and the COCO website.
One last note: if you’re looking for the latest results because you want to compare your results to SotA, great. However, if your goal is applying the “best” architecture on ImageNet to an industrial application using transfer learning, you should know (if you don’t already) that the latest architectures are worse, in terms of translation invariance, than the older ones. This is a risk if your dataset doesn’t have photographer bias of if you don’t have enough compute and data to retrain the architecture on a more useful image distribution. See the excellent preprint:
$\begingroup$
$\endgroup$
A "leaderboard" of sorts is maintained at this website, "Classification datasets results". The maintainers attempt to keep track of published results of various neural network architectures.
The leaderboard is not solely restricted to CNNs per se -- any network is admissible. But because all of the tasks tracked on the leaderboard are image tasks (as of this writing), it's likely that many of the networks will be CNNs since they are very effective at image tasks.
$\begingroup$
$\endgroup$
DenseNet is a generic successor to ResNet and achieves 3.46% error on CIFAR-10 and 17.18 on C-100. Compare to 3.47 and 24.28 mentioned on the leaderboard.
Shake-shake, Shake-drop and possibly other variants are regularization techniques which can be used any ResNet-like architectures, and achieves 2.86/2.31% error on C-10 and 15.85/12.19 on C-100 (shake-shake/shake-drop). These techniques work on only multi-branch architectures, which is why I mention them even though they are not strictly architectures in themselves.
Efficient Neural Architecture Search (using reinforcement learning to search for architectures) finds a network which achieves 2.89% error on C-10, using the Cutout regularization technique. Performance is 3.54% without cutout.
In summary: Dense Net and possibly some ENAS produced network may perform slightly better than ResNet, but the use of sophisticated regularization techniques makes the comparison admittedly difficult.
I don't know of any leaderboard which is really up to date, but typically any new paper which claims good or state-of-the-art performance on any task will have a fairly comprehensive results table comparing with previous results, which can be good way to keep track.
$\begingroup$
$\endgroup$
For checking state-of-the-art neural network architectures (and other machine learning models) in various application domains, there is now a page called paperswithcode.
