Comparison of statistical models with different target variable definitions Situation: I have a couple of statistical binary classification models (e.g. logistic regression, xgboost, random forest) and want to compare the model probabilities with each other. For simplicity, let's say each model were fitted on the same/similar training data. However, the target variable definition varies among the models.
So, e.g., there are target variables which are defined as e.g. customer falls into category XYZ in the next month. Another model defines the target variable as customer falls into category XYZ in next three month... or the month after next month.
Problem: Currently, I cannot compare probabilities from Model A with probabilities from Model B as the target variables are defined over different time horizons.
Question: Does there exist any kind of adjustment s.t. I can compare the models with each other (taking into account the time horizons of the target variable)?
EDIT: I have several product affinity models, each of them giving me a probability how likely it is that the customer will buy the according product. Since the targets are defined on varying time horizons I cannot directly compare the model probabilities with each other. So aim is to adjust model probabilities (taking into account time horizon) such that I can compare the product affinities with each other. (I don't ant to retrain all models again)
 A: Realistically, probably not without further assumptions. For those, starting from why you are comparing them would help:

*

*If you want to use them interchangeably, then you may only care about one task. In that case, maybe you could look at whether the models trained for tasks you don't care about still rank correctly for the task you care about by looking at some test set not used for training. E.g. if you really only care about 1 month predictions, does the model for the 3 month task at least rank observations correctly for the 1 month task? It's conceivable that a 3 month model could do better at ranking for the 1 month task than 1 month model, because it sees more events (and there's really not too much of a difference between events within 1 or within 2-3 months, it's really just a bit random when they happen, but the same records are at higher risk). In that case looking at ranking based metrics (e.g. area under the ROC or area under the PR curve) on one task could help.

*If your question is whether you should only predict for 1 month or 3 months (i.e. picking the task of interest), then comparing the two is really tough. Realistically, that would really need a business metric that incorporates the true business outcomes of interest and the way the models would really be used. You can then asess that against realistic data that was not used for training the models. E.g. if we keep forecasting once a week with a 1 month model vs. once a month with a 3 month model, and there is value $f(x)$ in identifying an event $x$ weeks ahead of time, a cost g(x) of wrongly forecasting an event, a cost $c$ of running the model, then what is the expected net outcome?

The other option is, of course, to re-train each model-type for each task.
