In my master thesis, I am researching on transfer learning on a specific use Case, a traffic sign detector implemented as a Single Shot Detector with a VGG16 base network for classification. The Research focuses on the problem of having a detector which is well trained and interfering on the traffic sign dataset it was trained for (I took the Belgium Traffic sign detection dataset) but when it Comes to use the detector in another Country (Germany, Austria, Italy, Spain,...) the traffic signs look more or less different which results in a certain unwanted loss. For an overview on this Topic, I would recommend the wikipedia article

~~~ the following section is about my Research questions ~~~

So having a couple of examples of traffic signs in the new Country, is it better to

  • fine tune the Network
  • Transfer learn the Network and freeze some of the convolution layers
  • (as a comparision) learn the new Country from scratch

Even the very first detector (the one trained from scratch on the comprehensive belgium dataset), does it have any advantage, to load pretrained weights from published model Zoos (for example VGG16/COCO) and then finetune/transferlearn based on this?

Now what am I asking here: I've implemented my detector not on my own, but based upon an original SSD port to Keras/Tensorflow (from here) and already trained it with different variations (Belgium from scratch, pretrained with MS COCO, Transfer to Germany, Convolution frozen, fine tuned to Germany) and after weeks of Training now I can say, that Belgium with random weights from scratch is converging fastest (after only 40 epochs/2 days my custom SSD loss function is down to a value of 3) while all other variations need much more time, more epochs and loss is never falling below a value of 9.

I also found pretrained weights for traffic sign classification with VGG16 which I thought should be the ideal base for transfer learning on this topic, but this detector was the worst performing so far (loss stagnated at 11, even when learning rate is changed and after 100 epochs it overfitted).

It seems, that transfer learning or fine tuning on these detectors doesn't have any advantage at all. It's likely that I am doing anything wrong or that I get the purpose of transfer learning wrong (I thought that it should speed up learning, as most layers aren't trainable and therefore no calculation is done)

I don't know if this is the proper platform for discussion on this topic, perhaps you know a slack or gitter channel which this belongs to. I just don't know if I am stuck, or I am just doing something horribly wrong.

  • 1
    $\begingroup$ I also have the same wondering. I've read some papers which mention that fine tuning is better than transfer learning. However when I tried to apply both of them on VGG16 for butterflies species classification, I found that fine tuning is worse than transfer learning! I am really confused. Did you reach a reasonable clarification? $\endgroup$
    – N0rA
    Commented Dec 6, 2018 at 11:57

2 Answers 2


Transfer learning is when a model developed for one task is reused to work on a second task. Fine-tuning is one approach to transfer learning where you change the model output to fit the new task and train only the output model.

In Transfer Learning or Domain Adaptation, we train the model with a dataset. Then, we train the same model with another dataset that has a different distribution of classes, or even with other classes than in the first training dataset).

In Fine-tuning, an approach of Transfer Learning, we have a dataset, and we use let's say 90% of it in training. Then, we train the same model with the remaining 10%. Usually, we change the learning rate to a smaller one, so it does not have a significant impact on the already adjusted weights. You can also have a base model working for a similar task and then freezing some of the layers to keep the old knowledge when performing the new training session with the new data. The output layer can also be different and have some of it frozen regarding the training.

In my experience learning from scratch leads to better results, but it is much costly than the others especially regarding time and resources consumption.

Using Transfer Learning you should freeze some layers, mainly the pre-trained ones and only train in the added ones, and decrease the learning rate to adjust the weights without mixing their meaning for the network. If you speed up the learning rate you normally face yourself with poor results due to the big steps in the gradient descent optimisation. This can lead to a state where the neural network cannot find the global minimum but only a local one.

Using a pre-trained model in a similar task, usually have great results when we use Fine-tuning. However, if you do not have enough data in the new dataset or even your hyperparameters are not the best ones, you can get unsatisfactory results. Machine learning always depends on its dataset and network's parameters. And in that case, you should only use the "standard" Transfer Learning.

So, we need to evaluate the trade-off between the resources and time consumption with the accuracy we desire, to choose the best approach.

  • 2
    $\begingroup$ Thanks for this clarification of methologies! You write about fine tuning that you split a dataset 90/10 % to first train and then finetune on the remaining 10%, and that this is an approach of transfer learning. But I thought that one specific aspect of transfer learning is, that the inital train dataset and the training dataset used to transfer learn must be from different domains (i.e. traffic signs from different countries). Isn't fine tuning just decreaseing the learning rate at later epochs to ensure that gradient descent is able to go down a minima it is zeroing in? $\endgroup$ Commented Jan 17, 2019 at 14:10
  • 2
    $\begingroup$ Fine-tune is transfer learning if the data on which the model is fine-tuned is of a different nature from the original data used to pre-train the model. So you can consider the case I said (90% to train, 10% fine-tune) as transfer learning by fine-tuning, where the 10% could have data from a different nature, or simply one different class. $\endgroup$ Commented Jan 17, 2019 at 15:03
  • $\begingroup$ I don't quite understand how dataset from different domain can be used to fine-tune a network because the distribution of the original data could be different compared to the new dataset. Could you please explain why you think it is beneficial ? $\endgroup$
    – VM_AI
    Commented Apr 2, 2020 at 12:34
  • $\begingroup$ @VM_AI The dataset should be from the same domain (assuming a domain as, for instance, car photos for a car classification system) and not from a different one, otherwise, as you said, it will not have any benefits from the fine-tune. $\endgroup$ Commented Apr 3, 2020 at 14:50
  • $\begingroup$ @VM_AI -- It's been shown (and, with NNs, there is an element of belief too) that the layers close to the input "learn" basic things about images such as edges etc. And this basic knowledge is "transferable" to another set of images from a different domain. As we go up the layers, the features "recognized" by the network may diverge from one set of images to the next. So, one can imagine keeping a lot of the pre-trained network if the new dataset is similar to the dataset used to pre-train the network -- or less if the datasets are different. The size of the new dataset is also a factor. $\endgroup$
    – user650654
    Commented Nov 11, 2021 at 18:21

Fine tuning, transfer learning, and learning from scratch are similar in that they are approaches to training a model on some data. But there are important differences.

Both fine tuning and transfer learning build on knowledge (parameters) an existing model has learned from previous data, while training from scratch does not build on the knowledge a model has previously learned.

Further, we can differentiate between fine tuning and transfer learning based on how exactly these two approaches build on existing knowledge. In transfer learning, you freeze the parameters taken from the existing model.

In fine tuning, you further train the parameters taken from the existing model.

Note that all three approaches can be combined during training. For example, you could create a new model by adding new layers to an existing model. Then, when training this new model, you could freeze only part of the existing model (the lower layers), fine tune the rest of the existing model (the upper layers), and train the new layers you added from scratch. Thus, transfer learning, fine tuning, and training from scratch can co-exist.

Also note, transfer learning cannot be used all by itself when learning from new data because of frozen parameters. Transfer learning needs to be combined with either fine tuning or training from scratch when learning from new data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.