0
$\begingroup$

I understand that in the case of the forward-pass the outputs are passed into a loss-function, rather than used to generate a prediction, but up until that point they are the same right?

I'm asking as I'm looking at two methods of using the VGG16 model. In one method the keras -model.predict- is used on the convolutional base of the VGG to generate features for new classifier layers which are then trained. In the second method the VGG base is frozen and new classifiers are trained on data passed I think into the frozen VGG base. To me the output from the frozen VGG base seems to be the same as the output from -model.predict-.

I know from the guidance they are very different methods [the second method is much more expensive], but I can't separate them conceptually.

The guidance:https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.3-using-a-pretrained-convnet.ipynb

Improve this question
$\endgroup$
1
  • $\begingroup$ One difference between the 2 methods i guess is that the 2nd method pass each image through the whole model multiple times as training progresses. Whereas the 1st method only sees each image once in the expensive VGG base? $\endgroup$
    – Hegerty
    Commented May 10, 2018 at 10:16

1 Answer 1

Reset to default
1
$\begingroup$

Terminology

A "forward pass" just means you put inputs in and propagate through the network, which in most implementations is a bunch of matrix multiplications and point-wise function applications. At the end, you get a set of outputs.

During training, the outputs are compared to what the answer should be, and the differences between the two--as measured by a loss function--are used to update model parameters, so now that input will cause the output to look a little bit more like the should-be answer.

Once the model is trained there is no backpropagation, no model-parameter-updating step, so there is no need to use the loss function. Actually, if you pass in some new input for which you don't know the answer, then there is nothing to compare the output against, so you don't even have all the information necessary to find a loss.

Keras

I interpret the methods you describe to be the same, at least by architecture: In both cases you use a convolutional network to get some simplified features out of the raw data and then pass those to a classifier that tells you something useful.

But there might be a difference in how they're trained. The word "frozen" implies that you train the CNN first and then stop updating its parameters while you train the classifier as a separate step. You can train the classifier via supervised learning because you presumably know what the classifications should be. But how do you train the CNN? One possibility is with an autoencoder architecture: A network tries to compress the input down to some smaller representation, and then a mirror network re-expands from that representation to the original space. (Imagine an input image compressed down to some key features and then propagated through a reverse network to yield a blurry or distorted version of the image.) The program can then use the difference between the true and re-constituted versions to decide how to update the network so that it can more accurately re-represent the input next time. Then take the first half (the compression part) of this autoencoder to be the CNN for your other problem. Voila.

But you say this is more expensive than the alternative: In principle it should be possible to backpropagate through all layers of both networks, so you could train both at once. This is likely what is happening in the method you say doesn't take as long.

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks. I believe the 'fast' method uses the prediction values from the VGG base [VGG16 without its classification layers] as features to pass into and train new custom classification layers. Only these new classification layers get trained, hence the speed. I don't think autoencoders play a part here, only because they do feature in a later chapter in Chollet's book and have not yet been introduced, but maybe? Chollet makes use of augmented images in method 2, I'm wondering if part of the expense is simply processing more images, but I don't really think that's the whole story?? $\endgroup$
    – Hegerty
    Commented May 11, 2018 at 10:23
  • $\begingroup$ Reading more deeply through that notebook you posted makes me realize "frozen" just means you're just using someone else's pretrained convnet and only replacing the classifier. This can work because the output of the convnet should be generic like "this part of the image has high frequency color variations", "this part of the image has a curve like this in it", "this part of the image is colored like this". So yes, this method should be faster. (You said this, the second, was slower.) $\endgroup$
    – Pavel Komarov
    Commented May 11, 2018 at 13:21
  • $\begingroup$ yes, I should have been clearer about 'frozen' == (trainable=False). But in my mind the predict operation in method1 doesn't train either [just a forward-pass], so I'd expect similar speeds. One advantage Chollet explicitly states fort method2 is it allows for the use of augmented images as data [but again I don't see why augmented images can't be passed into method1] $\endgroup$
    – Hegerty
    Commented May 11, 2018 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.