My understanding of fine-tuning is to take a pre-trained model trained on a similar but separate dataset and update the weights of a portion of the model on your dataset.
I'm not sure if this is called something entirely different, or if it's a terrible idea.. but say you have a dataset for a classification task where the data varies to a reasonable extent depending on some internal category, to the point that typically an individual model would be trained for each of these categories. Would it be a good idea (or possible) to train an initial model on the whole data, then fine-tune the model for each subset?
For example, some sort of data for a binary classification task involving five different subjects, where the expression of the positive category is generally similar but still differs on an individual basis. This type of data does not have any available pre-trained models, or additional freely available data that could be used instead.
I feel like I'm failing either with my choice of search terms, or I'm missing something that makes this a spectacularly bad idea, because I don't seem to be able to find anything about this sort of approach. I can clearly see overfitting being a large potential problem, but I still feel like I should be able to find something about this - even if it's just saying it's a bad idea. Is there a name for this kind of approach?