What does a "similar" dataset mean in the context of fine tuning a CNN? In https://arxiv.org/pdf/1809.09529.pdf it is said

If the new dataset is similar to
  original dataset, we expect higher-level features in the CNN
  to be relevant to this dataset. Thus, it is recommended to train
  a linear classifier on the CNN codes. Otherwise, if the new
  dataset is large then we might try to fine-tune through the full
  network. However if the new dataset is different from the
  original dataset and it has a small size, then it might work
  better to train a linear classifier from activations somewhere
  earlier in the network.

Fine tuning is quite popular in image classification problems where you take a model pre-trained on ImageNet and fine tune it to your dataset. But ImageNet classifies images on a very broad spectrum (1000 classes), so what does similar and different even mean precisely? How do I know if my dataset is similar to such a big and broad dataset? What does that mean?
 A: When speaking about the similarity for fine-tuning there are several points to consider:


*

*The classes of the 2 datasets - How similar are they? If your original dataset was of dogs and cats and now you move to a dataset of wolfs and tigers, than they are very similar and very deep features are still relevant (so you should only train 1 or 2 of the last layers). But if your new dataset contains cars and cows, you will need to train more layers  because less of them are relevant.

*The inner-variance of the classes - Even if your new dataset contains the same classes, you need to consider their inner-variance. A model that was trained to distinguish between frontal images of cat and dog faces will need to learn new features to recognize one of them from the side or back.

*The number of classes - A model that was trained on a dataset with very few classes compared to the new one, will need to learn more fine-grained features (so you will need to train more layers).

*The similarity/complexity of the task - If your model was trained for semantic segmentation and now you want to use it for classification, it won't be too hard. But going the other will be.   
In general, the deeper you go in the model, the more complex and specific are the features. The first layers are usually very simple and can generalize well between datasets and tasks (usually first layers are mostly about gradients, colors and textures).
Regarding on how to decide if your datasets are similar, this is a question that is usually answered subjectively, so you will have to answer it yourself. If you want something more official, you can try to analyze the 2 datasets in terms of the output of different layers in the model, comparing the mean and variance. At some point there should be some noticeable difference (note that I just made up this technique, so it might not help).
