Applying classification model when training and inference populations are different

Question

I am looking for ways of estimating or mitigating the risk of applying a classification model (say logistic regression for simplicity) in a certain population (the inference set) that is known to be different from the training population. We know that our metrics estimated in the test set are not directly applicable to the inference set since many of the features have different distributions. We are struggling to find ways of measuring how the model will be impacted and if/how the metrics measured can be translated to this inference set or which actions we should take before doing so. One idea was to build a simple distance based model to first filter cases that are close to the training set but it ends up filtering out too many cases, so I am open to suggestions :)

@Update

I have tried the following procedure to get more understanding of my data

1. Selected the N most important features of my model
2. Trained an IsolationForest with the inference set and predicted on the training set
3. Trained an IsolationForest with the training set and predicted on the inference set
4. Compared the % rejections between 2 and 3. The rejection rate is much lower on 3., which leads me to believe that the inference set is "contained" by the training set, despite the distributions being different.
5. In this scenario, it should be safe(ish) to apply my model

Can you please criticize the approach above?

Answer 1

I believe that you are right with regard to the concept out first identifying which samples can be considered to lie within your training distribution and which do not. Then you could simply assess how good the models work for samples considered inside the training distribution vs samples outside of it.

There is a whole research area considered with identifying such samples called Out-of-Distribution Detection. Here is a example of such.