0
$\begingroup$

In http://jmlr.org/proceedings/papers/v32/donahue14.pdf, it is stated:

Our top-performing method (based on validation accuracy) trains a linear SVM on DeCAF6

Can you delineate in a way hopefully tenable to relative laymen, how does one plug a layer of already trained neurons (DeCAF6 in this instance) into an SVM, both what exactly does this "integration" comprise, what is the overall flow, and some intuitions to help along the way?

Improve this question
$\endgroup$
2
$\begingroup$

The output of every layer of a neural network can be considered as features for a model (which usually consists of the following layers). Typically, neural networks use neurons in the output layer, but this may well be anything, including SVM as suggested in the paper.

In this sense, the deep network is used to construct features which allow linear separability by the SVM. That said, I don't see the appeal of using a linear SVM as a drop-in replacement for a typical output layer, as they are almost equivalent.

Improve this answer
$\endgroup$
5
  • $\begingroup$ When saying the output of each layer can be seen as features, do you mean this per input fed to the layer? $\endgroup$
    – matt
    Aug 10 '14 at 15:41
  • 2
    $\begingroup$ Yes. You can use a neural network to construct features from an image, $f:\mathbb{R}^{n\times n}\mapsto\mathbb{R}^m$. Neural networks are very good at representational learning. $\endgroup$
    – Jessica Collins
    Aug 10 '14 at 16:33
  • $\begingroup$ So does the original sentence in the article mean that the layer carved out of the original neural network had been sit, with it's learnt weights now fixed, on top an SVM, and then this amalgamated formation had been trained (by funneling inputs through the neural layer, into the SVM, rather than just training the bare SVM) ? $\endgroup$
    – matt
    Aug 10 '14 at 20:52
  • 1
    $\begingroup$ Yes. This is mentioned explicitly in the caption of Figure 4: Our result from the training protocol/classifier combination with the best validation accuracy SVM with Layer 6 (+ dropout) features – is shown in bold, e.g. an SVM which uses the outputs of layer 6 and dropout features as input. $\endgroup$
    – Marc Claesen
    Aug 10 '14 at 21:02
  • $\begingroup$ Thanks guys, yes it is explicit there assuming one already knows how this is typically done. All connects now. Thanks so much! $\endgroup$
    – matt
    Aug 10 '14 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.