I have two datasets of MRI images: a larger one of Altzheimer's paitents (AD), which is about 3 times the size of a smaller dataset of brain tumor paitents (BT). My aim is to make use of the AD data with some form of transfer learning to help with the classification of brain tumor images.
Question: Would it be viable to train a feature extractor unsupervised (likely a stacked convolutional autoencoder) using only the AD data, and then use the encoder as the first few layers of a new network which has fully-connected layers trained on the BT data?
My thinking is that it may be possible to have the autoencoder 'learn' the general features of the brain from the AD data, and then use this as a basis for a BT network. I have not seen this done before (which maybe for a good reason).
Perhaps there is some issue with overfitting/general transferability between the datasets. Would fine-tuning the encoder when the fully-connected layers are trained possibly prevent this?