1
$\begingroup$

I want to use a pretrained neural network for two similar (but not identical) classification problems. Let's say I want to use AlexNet for image classification, where in problem A I am interested in classifying images in two classes, and in problem B I am interested in classifying in six classes.

I then create two instances of a pre-trained AlexNet and add a last layer with size according to the number of classes I want to classify. I then train these two NNs with a reduced, specific dataset.

What happens to the weights of these NNs?

I guess that the first few layers will detect more basic aspects, and thus the weights there won't change (therefore the two NNs will be equal in this part), and the last layers will have different weights because they are being fine-tuned for my applications (and therefore my two NNs would differ in this part). Is this correct? If so, how can I know where the point of divergence is between the two NNs that I have?

$\endgroup$
0
$\begingroup$

The weights in the lower layers of the network will also change, but not as drastically as those in the higher layers. (Gradients are propagated all the way to the base layers during backpropagation.) Adapting higher layers (and their resulting more task-specific representations) has greater impact in transferring to similar tasks.

This behavior has been studied in a different field (natural language processing) by Thompson et al. (2018). The work examines the effect of freezing particular network weights during training, to determine which ones are most crucial and able to adapt during transfer learning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.