I have a dataset with 119 features and 228 samples. I have trained my classifier using a particular set of labels (0 and 1). Now I want to test how the model is performing to external data. There is not a single dataset out there with the same labels (0 and 1). But my supervisor really wanted to look for external validation of our model. That prompted me to look of other test datasets with the same feature set but a different label. I found one such dataset with 55 features overlapping with my train set. But the labels for the new test set are 67% correlated with my train labels. Should I test my model on this new data?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
you can, but it's not really testing the generalization of your model since you are not using it for the same problem.
People sometimes do it, for example, someone might fit a model to classify healthy and depressed people and then show that it also classify healthy and anxious people but not healthy and schizophrenic people. This is then followed by speculations about why is it or bold claims about some kind of magical transdiagnostic effects or other speculations about for what purposes that model can be used.
-
$\begingroup$ Thanks for the reply. But I am wondering if there is any natural limitation to what my model can achieve in this case. First, only 55 features overlap between the two datasets and then it isn't a 100% correlation between the labels. Any suggestions? $\endgroup$ Commented Oct 30, 2019 at 14:08
-
1$\begingroup$ you can create a model using only the overlapping features. There is no workaround for different labels, then you would just have to use your domain knowledge to interpret the results. If the labels are completely unrelated, then there is nothing you can do about it. $\endgroup$– rep_hoCommented Oct 30, 2019 at 14:26