# Using likelihood to compare model accuracy on two different data sets

I have a model that produces conditional probabilities, $p(Y|X)$, where $Y$ is either 0 or 1, and $X$ is just some random variable.

I have two different data sets $Z_1, Z_2$ consisting of pairs of outcomes of Y and X:

$Z_1 = \{(x^1_1, y^1_1), (x^1_2, y^1_2), (x^1_3, y^1_3), \ldots, (x^1_n, y^1_n)\}$

and

$Z_2 = \{(x^2_1, y^2_1), (x^2_2, y^2_2), (x^2_3, y^2_3), \ldots, (x^2_m, y^2_m)\}$.

The distributions of $X$ conditional on residing in these data sets are different.

I now wish to compare how well the model fits the data, in the sense of log likelihood, on these sets.

I realize that I cannot simply compare the empirical average of the log likelihoods; $\frac{1}{n} \sum_{i=1}^n log(p(y^1_i | x^1_i))$ and $\frac{1}{m} \sum_{i=1}^m log(p(y^2_i | x^2_i))$.

To see that this would be wrong, assume that the "true" probabilities on the datasets are constant, $p_1$ and $p_2$ say, and the the model produces exactly these constant probabilities on the relevant data sets.

Then the expected log-likelihoods would be $p_i \cdot log(p_i) + (1-p_i) \cdot log(1- p_i)$, for $i \in \{1, 2\}$, which would give different values if the $p_i$ are unequal although the model is "perfect" on both sets, and we would erroneously deduce that it's better on one of them.

One thought I had was the following:

For each set, I can compute the constant probability maximizing the likelihood; this will just yield $\frac{1}{n} \cdot \sum_{i=1}^n y^1_i$ and $\frac{1}{m} \cdot \sum_{i=1}^m y^2_i$, respectively.

For each data set, I now subtract the average log-likelihood of the optimal constant model from the ditto of the actual model, to obtain a measure of the relative improvement the conditional model gives as compared to the constant model that best fits the data set.

In my particular case, I get that on $Z_1$ the conditional model performs better than the optimal constant model, while for $Z_2$ the converse relation holds.

It seems to me, however, that this approach might be problematic as well. Perhaps the optimal constant model on $Z_2$ is closer to the "true" distribution, in which case it is harder to "beat".