How to compare log-loss across similar classification models with different baseline probabilities? Suppose I have two datasets, A and B, which share a feature vector $X$ but have different units of analysis (e.g. people from two different countries). I have trained classifiers with the same model architecture and features on these two datasets to predict some boolean variable $y$.
Now I want to compare the two classifiers to make some claim like: "this particular model specification performs better on dataset A than on dataset B." My chosen metric is log-loss, as I care about the probability classification.
My intuition is that comparing log-loss directly across the two models doesn't make sense, because the distribution of positives and negatives are different (i.e. $p_{y,A} \neq p_{y,B}$). That said, is there a way to use log-loss to compare performance on the two datasets? For example, some kind of normalized log-loss?
 A: You might be interested in something like $R^2$, where we compare model performance to the performance of a naïve model that always guesses the mean.
$$
R^2 = 1 - \dfrac{
\sum\big(
y_i - \hat y_i
\big)^2
}{
\sum\big(
y_i - \bar y
\big)^2
}
=
1 - \dfrac{
\text{Model square loss}
}{
\text{
Naïve model square loss
}
}
$$
Apply this idea to log loss instead of square los for so-called McFadden's $R^2$.
$$
R^2_{McFadden} =
1 - \dfrac{
\text{Model log loss}
}{
\text{
Naïve model log loss
}
}
$$
UCLA has a page with a number of other $R^2$-style metrics for models that predict probabilities.
A: If you have enough data, you could subsample from some of the classes in order to obtain the same positive/negatives ratio in the two datasets. This should make your log-loss somewhat comparable in the two datasets.
A: The log loss of a set of predictions can be described as,
$$
-L = \sum_{y_i=1} \log p_i + \sum_{y_i = 0} \log (1 - p_i).
$$
The two main things this is sensitive to, in the data, is number of samples and proportion of $1$s in the data. For proportion of $1$s, for example, if your model is much better at correctly classifying $0$s than $1$s, the dataset with a larger portion of $0$s would have the advantage. Also, $-L$ would simply be a larger number for a large number of samples.
Obviously you could account for the sample difference by simply dividing by $N$, the number of samples in the dataset. What this would give you is, essentially, the log of the geometric mean of the probability of a sample target value given your prediction. That is because the geometric mean is just $\text{likelihood}^{1/N}$, so the log of that is simply $\frac{1}{N} \text{log-likelihood}.$
To account for the ratio, you can simply weight the two terms in the equation for $-L$ equally. So instead of taking the above suggestion, which would basically be,
$$
\frac{-L}{N} = \frac{1}{N} \left( \sum_{y_i=1} \log p_i + \sum_{y_i = 0} \log (1 - p_i) \right),
$$
you could do,
$$
-L_{\text{weighted}} = \frac{1}{2} \left( \frac{1}{N_1} \sum_{y_i=1} \log p_i + \frac{1}{N_0}  \sum_{y_i = 0} \log (1 - p_i) \right),
$$
where $N_1 = \sum_i y_i$ and $N_0 = \sum_i (1 - y_i)$.
What this is is the negative log of the equally-weighted geometric mean of the geometric mean of the likelihood for $y=1$ samples and the geometric mean of the likelihood of $y=0$ samples. That is,
$$
-L_{\text{weighted}} = - \log \left\{ \left( \text{likelihood}_1 ^\frac{1}{N_1} \text{likelihood}_0^\frac{1}{N_0} \right)^\frac{1}{2} \right\}
$$
where $\text{likelihood}_1$ is the likelihood of observing the $y=1$ portion of the data given your probability predictions for those, and similar definition for $\text{likelihood}_0$.
To be clear, there are good metrics to use besides log-loss, for comparing classification on different data sets. Metrics such as area under ROC curve, TPR, TNR, etc. But if you're set on using log-loss, and you don't want to give one set an advantage over another based on sample size or label proportion, then you could use my suggestion.
