3
$\begingroup$

Suppose I have two datasets, A and B, which share a feature vector $X$ but have different units of analysis (e.g. people from two different countries). I have trained classifiers with the same model architecture and features on these two datasets to predict some boolean variable $y$.

Now I want to compare the two classifiers to make some claim like: "this particular model specification performs better on dataset A than on dataset B." My chosen metric is log-loss, as I care about the probability classification.

My intuition is that comparing log-loss directly across the two models doesn't make sense, because the distribution of positives and negatives are different (i.e. $p_{y,A} \neq p_{y,B}$). That said, is there a way to use log-loss to compare performance on the two datasets? For example, some kind of normalized log-loss?

$\endgroup$
1
  • $\begingroup$ I'm not sure that the phrase you suggest ("this particular model specification performs better on dataset A than on dataset B") really expresses what you want to express. If you define better in terms of log-loss achieved, then there is no correction to make, because you are just saying that one achieves a better log loss. $\endgroup$
    – Ryan Volpi
    Mar 19, 2021 at 15:21

3 Answers 3

3
$\begingroup$

You might be interested in something like $R^2$, where we compare model performance to the performance of a naïve model that always guesses the mean.

$$ R^2 = 1 - \dfrac{ \sum\big( y_i - \hat y_i \big)^2 }{ \sum\big( y_i - \bar y \big)^2 } = 1 - \dfrac{ \text{Model square loss} }{ \text{ Naïve model square loss } } $$

Apply this idea to log loss instead of square los for so-called McFadden's $R^2$.

$$ R^2_{McFadden} = 1 - \dfrac{ \text{Model log loss} }{ \text{ Naïve model log loss } } $$

UCLA has a page with a number of other $R^2$-style metrics for models that predict probabilities.

$\endgroup$
1
$\begingroup$

The log loss of a set of predictions can be described as, $$ -L = \sum_{y_i=1} \log p_i + \sum_{y_i = 0} \log (1 - p_i). $$ The two main things this is sensitive to, in the data, is number of samples and proportion of $1$s in the data. For proportion of $1$s, for example, if your model is much better at correctly classifying $0$s than $1$s, the dataset with a larger portion of $0$s would have the advantage. Also, $-L$ would simply be a larger number for a large number of samples.

Obviously you could account for the sample difference by simply dividing by $N$, the number of samples in the dataset. What this would give you is, essentially, the log of the geometric mean of the probability of a sample target value given your prediction. That is because the geometric mean is just $\text{likelihood}^{1/N}$, so the log of that is simply $\frac{1}{N} \text{log-likelihood}.$

To account for the ratio, you can simply weight the two terms in the equation for $-L$ equally. So instead of taking the above suggestion, which would basically be, $$ \frac{-L}{N} = \frac{1}{N} \left( \sum_{y_i=1} \log p_i + \sum_{y_i = 0} \log (1 - p_i) \right), $$

you could do,

$$ -L_{\text{weighted}} = \frac{1}{2} \left( \frac{1}{N_1} \sum_{y_i=1} \log p_i + \frac{1}{N_0} \sum_{y_i = 0} \log (1 - p_i) \right), $$ where $N_1 = \sum_i y_i$ and $N_0 = \sum_i (1 - y_i)$.

What this is is the negative log of the equally-weighted geometric mean of the geometric mean of the likelihood for $y=1$ samples and the geometric mean of the likelihood of $y=0$ samples. That is,

$$ -L_{\text{weighted}} = - \log \left\{ \left( \text{likelihood}_1 ^\frac{1}{N_1} \text{likelihood}_0^\frac{1}{N_0} \right)^\frac{1}{2} \right\} $$ where $\text{likelihood}_1$ is the likelihood of observing the $y=1$ portion of the data given your probability predictions for those, and similar definition for $\text{likelihood}_0$.

To be clear, there are good metrics to use besides log-loss, for comparing classification on different data sets. Metrics such as area under ROC curve, TPR, TNR, etc. But if you're set on using log-loss, and you don't want to give one set an advantage over another based on sample size or label proportion, then you could use my suggestion.

$\endgroup$
0
$\begingroup$

If you have enough data, you could subsample from some of the classes in order to obtain the same positive/negatives ratio in the two datasets. This should make your log-loss somewhat comparable in the two datasets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.