33
$\begingroup$

I have seen people have put a lot of efforts on SVM and Kernels, and they look pretty interesting as a starter in Machine Learning. But if we expect that almost-always we could find outperforming solution in terms of (deep) Neural Network, what is the meaning of trying other methods in this era?

Here is my constraint on this topic.

  1. We think of only Supervised-Learnings; Regression, and Classification.
  2. Readability of the Result is not counted; only the Accuracy on the Supervised-Learning Problem counts.
  3. Computational-Cost is not in consideration.
  4. I am not saying that any other methods are useless.
$\endgroup$
  • 3
    $\begingroup$ Any constraints on the amount of training data available? $\endgroup$ – Jake Westfall Feb 20 '17 at 0:53
  • 1
    $\begingroup$ I haven't done it, but I expect you'd have one hell of a hard time training a neural network to do e.g. a singular-value decomposition on a non-trivially-sized matrix (say, rank > 10). $\endgroup$ – user541686 Feb 20 '17 at 3:21
  • 1
    $\begingroup$ Google Translate now uses neural networks, and now produces more curious fails for a Brazilian town name where a word-for-word use of a dictionary would have been much better $\endgroup$ – Henry Feb 20 '17 at 15:21
  • $\begingroup$ I haven't ever heard of deep learning for use of matrix completion (although using matrix completion before deep learning is common practice). You could argue this could be a computational-cost issue, but it's also worth noting that I don't know if all the computers in the world could do deep learning matrix completion with, say, the netflix problem. $\endgroup$ – Cliff AB Feb 20 '17 at 20:58
  • $\begingroup$ @CliffAB: (tongue halfway in cheek...) it's probably worth noting that they might not be able to, but I'm not sure it's worth noting that you don't know if they are able to ;) $\endgroup$ – user541686 Feb 21 '17 at 3:35
31
$\begingroup$

Here is one theoretical and two practical reasons why someone might rationally prefer a non-DNN approach.

  1. The No Free Lunch Theorem from Wolpert and Macready says

    We have dubbed the associated results NFL theorems because they demonstrate that if an algorithm performs well on a certain class of problems then it necessarily pays for that with degraded performance on the set of all remaining problems.

    In other words, no single algorithm rules them all; you've got to benchmark.

    The obvious rebuttal here is that you usually don't care about all possible problems, and deep learning seems to work well on several classes of problems that people do care about (e.g., object recognition), and so it's a reasonable first/only choice for other applications in those domains.

  2. Many of these very deep networks require tons of data, as well as tons of computation, to fit. If you have (say) 500 examples, a twenty layer network is never going to learn well, while it might be possible to fit a much simpler model. There are a surprising number of problems where it's not feasible to collect a ton of data. On the other hand, one might try learning to solve a related problem (where more data is available), use something like transfer learning to adapt it to the specific low-data-availability-task.

  3. Deep neural networks can also have unusual failure modes. There are some papers showing that barely-human-perceptible changes can cause a network to flip from correctly classifying an image to confidently misclassifying it. (See here and the accompanying paper by Szegedy et al.) Other approaches may be more robust against this: there are poisoning attacks against SVMs (e.g., this by Biggio, Nelson, and Laskov), but those happen at train, rather than test time. At the opposite extreme, there are known (but not great) performance bounds for the nearest-neighbor algorithm. In some situations, you might happier with lower overall performance with less chance of catastrophe.

$\endgroup$
  • $\begingroup$ I agreed everything you say. But the problem is about "computational issues ignored". That means the OP assumes, you'd have infinite samples and infinite computing resources. $\endgroup$ – HelloWorld Feb 20 '17 at 1:51
  • 17
    $\begingroup$ Infinite computation != infinite samples. For example, I have access to an astonishingly large cluster for processing data. However, the lab experiments we do to actually acquire some of that data are difficult, slow, and time-consuming (on the order of hours to days for a single data point) and all the computation in the world won't help that along. $\endgroup$ – Matt Krause Feb 20 '17 at 1:54
  • 2
    $\begingroup$ SVMs with any given feature extractor are likely just as vulnerable to adversarial inputs as CNNs – it's just harder to find them, because we don't have easily available gradients of the feature extraction layers. $\endgroup$ – Dougal Feb 20 '17 at 10:41
  • 1
    $\begingroup$ A recent and interesting real-life example of the problem @MattKrause gives and an attempt to circumnavigate it using transfer learning is presented in Sim-to-Real Robot Learning from Pixels with Progressive Nets $\endgroup$ – HBeel Feb 20 '17 at 11:10
  • $\begingroup$ @Dougal, I also wonder if it matters that the DNN-feature-extractors are learned, while the SVM ones are (usually) hand-made and correspond to features that humans notice. Part of what makes the panda example so insidious is the imperceptible difference between the adversarial example and the regular one. $\endgroup$ – Matt Krause Feb 20 '17 at 16:57
24
$\begingroup$

Somewhere on this playlist of lectures by Geoff Hinton (from his Coursera course on neural networks), there's a segment where he talks about two classes of problems:

  1. Problems where noise is the key feature,
  2. Problems where signal is the key feature.

I remember the explanation that while neural nets thrive in this latter space, traditional statistical methods are often better suited to the former. Analyzing high-res digital photographs of actual things in the world, a place where deep convolutional nets excel, clearly constitutes the latter.

On the other hand, when noise is the dominant feature, for example, in a medical case-control study with 50 cases and 50 controls, traditional statistical methods may be better suited to the problem.

If anybody finds that video, please comment and I'll update.

$\endgroup$
  • $\begingroup$ Excellent answer. Exactly why we turn to deep learning for things we can already do (like recognize images and write text) but may turn to other models for things that may be intuitively difficult. $\endgroup$ – Mustafa S Eisa Feb 21 '17 at 20:45
  • $\begingroup$ I personally take this answer in my greatest interest. Thank you so much for the answer. $\endgroup$ – Robin Feb 22 '17 at 9:01
13
$\begingroup$

Two linearly perfected correlated variables. Can deep-network with 1 million hidden layers and 2 trillion neutrons beat a simple linear regression?

EDITED

In my experience, sample collection is more expensive than computation. I mean, we can just hire some Amazon instances, run deep learning training and then come back a few days later. The cost in my field is about $200 USD. The cost is minimal. My colleagues earn more than that in a day.

Sample collection generally requires domain knowledge and specialized equipments. Deep learning is only suitable for problems with cheap and easy access data set, such as natural language processing, image processing and anything that you can scrape off from the Internet.

$\endgroup$
  • 1
    $\begingroup$ Of course any MLE method will out-perform deep learning conditional on the generating model meeting the assumptions of the MLE. However, this is never happens on real data, or at least for any interesting problems (i.e. not predicting outcome of coin flip). So I think the OP is asking for examples involving real questions of interest with real data. $\endgroup$ – Cliff AB Feb 20 '17 at 20:48
  • $\begingroup$ It is a very nice answer. You have offered a very intuitive, and realistic viewpoint. Thank you so much. $\endgroup$ – Robin Feb 22 '17 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.