2
$\begingroup$

We have an "old-style" image recognition system that is achieving 90-95% recognition accuracy. We need to improve that to 95-98% accuracy, preferably better. If we embark on a convolutional neural network R&D program, how confident can we be that we will achieve these targets? What factors point towards success/failure?

Context: by "old-style" I mean a system which hand-tailors features, makes measurements, and submits those measurements to a simple non-black-box classifier (e.g. decision tree, nearest neighbour). We didn't think that NNs as they existed 10 years ago (i.e. MLP/backpropagation) would achieve 90%+ accuracy on our problem, but that if they "almost" did, we could easily struggle to understand why they were falling short due to their black-box nature, and we might therefore waste a lot of time trying to improve them. They were a gamble we were unwilling to take. How can we tell if convnets are or are not a gamble too?

Recognising also that 90% accuracy is terrible in some contexts, and terrific in others, our problem might as well be farmyard animal recognition. It is characterised by:

  • a dozen or so classes (more or less depending on how specific you want to be, e.g. "cow" and "dog" versus "Jersey", "Holstein", "Hereford" and "Border Collie", "Old English" etc)
  • wide variation in training samples from some classes, but not others (e.g. "all chickens look the same to me, but no two cows are ever the same")
  • different data is relevant to classifying different classes (e.g. what distinguishes two-legged animals is different from what distinguishes four-legged ones)

How can one go about deciding whether convolutional neural networks represent an investment of several weeks/months, or a likely waste of that time, in such a context?

Alternatively: I may not be allowed/encouraged to seek opinions here, but if I need to seek opinions on this matter, what specific questions should I be asking?

$\endgroup$

1 Answer 1

2
$\begingroup$

You can't be sure that they will be better then your classifier, however given the fact that there are multiple examples of neural networks achieving very high accuracy and that companies like Google use and develop them for such purposes, this seems likely. You can google for "Convolutional Neural Network image classification accuracy", to find multiple tutorials that with toy models easily achieve >80% accuracy with little or no tuning, while state-of-art algorithms can be much better, as the one described in the (randomly chosen) paper by Hasanpour et al. who describe algorithms that achieve >99% accuracy with some datasets. Notice however that this highly depends on the model you choose and the data you are dealing with (you can always find data where state-of-art algorithms will fall).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.