I have a model that produces conditional probabilities, $p(Y|X)$, where $Y$ is either 0 or 1, and $X$ is just some random variable.

I have two different data sets $Z_1, Z_2$ consisting of pairs of outcomes of Y and X:

$Z_1 = \{(x^1_1, y^1_1), (x^1_2, y^1_2), (x^1_3, y^1_3), \ldots, (x^1_n, y^1_n)\}$


$Z_2 = \{(x^2_1, y^2_1), (x^2_2, y^2_2), (x^2_3, y^2_3), \ldots, (x^2_m, y^2_m)\}$.

The distributions of $X$ conditional on residing in these data sets are different.

I now wish to compare how well the model fits the data, in the sense of log likelihood, on these sets.

I realize that I cannot simply compare the empirical average of the log likelihoods; $\frac{1}{n} \sum_{i=1}^n log(p(y^1_i | x^1_i))$ and $\frac{1}{m} \sum_{i=1}^m log(p(y^2_i | x^2_i))$.

To see that this would be wrong, assume that the "true" probabilities on the datasets are constant, $p_1$ and $p_2$ say, and the the model produces exactly these constant probabilities on the relevant data sets.

Then the expected log-likelihoods would be $p_i \cdot log(p_i) + (1-p_i) \cdot log(1- p_i)$, for $i \in \{1, 2\}$, which would give different values if the $p_i$ are unequal although the model is "perfect" on both sets, and we would erroneously deduce that it's better on one of them.

One thought I had was the following:

For each set, I can compute the constant probability maximizing the likelihood; this will just yield $\frac{1}{n} \cdot \sum_{i=1}^n y^1_i$ and $\frac{1}{m} \cdot \sum_{i=1}^m y^2_i$, respectively.

For each data set, I now subtract the average log-likelihood of the optimal constant model from the ditto of the actual model, to obtain a measure of the relative improvement the conditional model gives as compared to the constant model that best fits the data set.

In my particular case, I get that on $Z_1$ the conditional model performs better than the optimal constant model, while for $Z_2$ the converse relation holds.

It seems to me, however, that this approach might be problematic as well. Perhaps the optimal constant model on $Z_2$ is closer to the "true" distribution, in which case it is harder to "beat".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.