I am working with a classification task in the field of biology. For this task, I am trying to assess whether or not newly observed data points fit into one of two classes using a logistic regression model trained on data with well-defined labels (i.e. the new data is not used to train the model at all, it is just used to derive a mapping onto one of two classes via the logistic model). I'd like to assess whether the new datapoints fit into these classes well or not (e.g. the mean logistic predictions of new datapoints is far enough from 0.5). (A) How would one do this in a statistically rigorous manner? I have several unlabelled datasets per subject that I'd like to run and produce these statistics for. (B) I am also curious as to how I would summarize such statistics across subjects (each subject has the same set of unlabelled datasets). Any pointers regarding A and B?
Here are a few ideas I had. I'm a little unsure of the difference between A & B and the form of your data from your description.
If you knew that your test dataset had roughly the same number of cases/controls as the data you use for training it may make sense to perform a kolmogorov-smirnov test on the (1) cross validated prediction probabilities and the (2) test set prediction probabilities. This would get you a statistic on how comparable the probability distributions are between your training and test set.
Another option that wouldn't have the case/control assumption might be to do an analysis involving permutation. For this, you could randomly shuffle your features between cases, thus destroying the relationships between variables within each observation. When applying your model to this permuted dataset you would expect many values to be close to 0.5. You could then train a logistic regression model that is predicting whether or not a prediction probability of an individual case (y) is from a permuted or not permuted sample (x). The p-value/test statistic from the binary permutation feature could be compared between your unlabeled data and with the same procedure performed in cross-validation on your training dataset.