In https://arxiv.org/pdf/1809.09529.pdf it is said

If the new dataset is similar to original dataset, we expect higher-level features in the CNN to be relevant to this dataset. Thus, it is recommended to train a linear classifier on the CNN codes. Otherwise, if the new dataset is large then we might try to fine-tune through the full network. However if the new dataset is different from the original dataset and it has a small size, then it might work better to train a linear classifier from activations somewhere earlier in the network.

Fine tuning is quite popular in image classification problems where you take a model pre-trained on ImageNet and fine tune it to your dataset. But ImageNet classifies images on a very broad spectrum (1000 classes), so what does similar and different even mean precisely? How do I know if my dataset is similar to such a big and broad dataset? What does that mean?


When speaking about the similarity for fine-tuning there are several points to consider:

  • The classes of the 2 datasets - How similar are they? If your original dataset was of dogs and cats and now you move to a dataset of wolfs and tigers, than they are very similar and very deep features are still relevant (so you should only train 1 or 2 of the last layers). But if your new dataset contains cars and cows, you will need to train more layers because less of them are relevant.

  • The inner-variance of the classes - Even if your new dataset contains the same classes, you need to consider their inner-variance. A model that was trained to distinguish between frontal images of cat and dog faces will need to learn new features to recognize one of them from the side or back.

  • The number of classes - A model that was trained on a dataset with very few classes compared to the new one, will need to learn more fine-grained features (so you will need to train more layers).

  • The similarity/complexity of the task - If your model was trained for semantic segmentation and now you want to use it for classification, it won't be too hard. But going the other will be.

In general, the deeper you go in the model, the more complex and specific are the features. The first layers are usually very simple and can generalize well between datasets and tasks (usually first layers are mostly about gradients, colors and textures).

Regarding on how to decide if your datasets are similar, this is a question that is usually answered subjectively, so you will have to answer it yourself. If you want something more official, you can try to analyze the 2 datasets in terms of the output of different layers in the model, comparing the mean and variance. At some point there should be some noticeable difference (note that I just made up this technique, so it might not help).

  • $\begingroup$ Thank you for your contribution, this was very insightful! I am working on binary classification of skin samples from isic-archive.com as either benign or malignant (2 very specific classes). I am considering fine-tuning VGG16 pre-trained on ImageNet (1000 very broad classes). The classes are quite different but the task is the same and the number of classes is much lower than the pretrained model's. Would you say they are similar or not? $\endgroup$ – fabiomaia Jan 21 '19 at 16:37
  • 1
    $\begingroup$ I'd say the VGG16 will be a clear overkill for this binary task. My suggestion is to start from a much simpler model, for instance the "All-Convolutional Net" trained on the CIFAR10. $\endgroup$ – Mark.F Jan 22 '19 at 13:07
  • $\begingroup$ Why a simpler (i.e. less parameters?) model and why a 10-class dataset? Note that CIFAR10 uses 32x32 images whereas the ISIC Archive dataset's images are very high resolution. $\endgroup$ – fabiomaia Jan 22 '19 at 13:55
  • $\begingroup$ Simpler model because it has considerably less parameters (around 100 times less: 138M compared with 1.4M). $\endgroup$ – Mark.F Jan 24 '19 at 7:41
  • $\begingroup$ And there is no real reason to choose 10 classes rather than 1000, other than the fact that it is closer to the 2 classes in your case. Eventually your model should be relatively light and simple. $\endgroup$ – Mark.F Jan 24 '19 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.