Develop granularity-invariant criteria for comparison of logistic (binomial) models

Question

I have a model with logistic (binomial) likelihood, with number of successes and failures as a response variable. I am comparing various models, which can be of different granularity. Different granularity means that the binomial observations can be either:

• grouped together (successes and failures summed up) for each site, or
• evaluated separately for each visit (there can be multiple visits to each site).

So, I am looking for model quality criteria, which wouldn't change with the site/visit granularity; i.e. which would produce the same result regardless of how the binomial observations are grouped.

I developed bunch of model comparison criteria, but as you can see below, apart from the AUC, all of them change with granularity. Here below is the evaluation of a single model using different criteria - first column shows the site-level granularity, second column the visit-level granularity:

                  per_site  per_visit
AUC_1h          0.97175420 0.97175420
AUC_1h_weighted 0.97033082 0.97033082
R2_avgScore     0.49352020 0.42906301
R2_dev          0.68408469 0.53648654
R2_LR           0.62293855 0.53648654

R2_dev is pseudo $R^2$ based on deviance, R2_LR is based on likelihood, McFadden’s - see definitions here.

The problem with binomial likelihood:

$$\prod_{i}{n_i \choose x_i}p_i^x(1-p_i)^{n_i-x_i}$$

is that it contains the binomial coefficient ${n_i \choose x_i}$, which is the (only) term which depends on the granularity.

Since I don't want to stick just to AUC, I tried to look for other pseudo-R-squared methods for one which would be granularity-invariant. The Cox & Snell did look promissing:

because the binomial coefficients would cancel each other out in the fraction. However, there are two problems with this:

1. It needs a modification: $N$ needs to be set up so that it is granularity invariant. So instead of putting $N$ as number of records, one would put $N$ as the total sum of all successes and failures (which doesn't change with granularity). Would that make sense? Or is there any conceptual problem with this modification?

2. The maximum of this criteria is not one, which makes it difficult to interpret. This is addressed by Nagelkerke / Cragg & Uhler’s pseudo R-squared:

but here again, the denominator will ruin the granularity-independence again, as it depends on the binomial coefficient.

So how to address this?

1. Is there a way to reasonably modify Cox & Snell? (See the 2 points above)
2. Or would it make sense to just use all of these likelihood-based criteria, and just calculate the likelihood without the binomial coefficients? Would that make sense?
3. Is there another reasonable, granularity-invariant criteria?
4. Is my way of thinking alright, or is it conceptually broken (for example because the granularity is so important, that it doesn't make sense to look for granularity invariant criteria)? Why?

Answer 1

Why do you get differences in $R^2$

The $R^2$ is a bit meaningless here. It is an indicator of the relative variance of the data and the variance of the model. If this ratio is closer to 1 then the model is considered better because the estimates match the data better.

However, if you look at per visit instead of per site then the variance of the data will be larger because there is variance within a site that is not taken into account when you only look at 'per site' averages.

• You can accurately predict the mean of a site and even obtain very high $R^2$ values given enough data per site.

• But, within the site there will always be variation; for a single visit you only get a success or a failure, and there is no half successful visit. The single visits, being limited to either 0 or 1 (or positive/negative, success/failure, etc.), will necessarily have a discrepancy with the estimated $p_i$ values.

The model is only predicting the per site $p_i$ values and not the per visit single outcomes. So when you compare 'per site' then the model may have a higher $R^2$ than when you compare 'per visit'.

Differences in likelihood 'per site vs. per visit'

It will be better to use methods that are based on likelihood. You still get differences, but they are not meaningful for comparison.

The probability of the observations for the grouped cases is

$$P(x_i \vert p_i) = \prod_{i}{n_i \choose x_i}p_i^{x_i}(1-p_i)^{n_i-x_i}$$

and for the separate observations

$$P(x_{ij} \vert p_i) = \prod_{ij} p_i^{x_{ij}}(1-p_i)^{1-x_{ij}} = \prod_{i}p_i^{x_i}(1-p_i)^{n_i-x_i}$$

Where the last equality is made by grouping all of the terms in the same group. The difference is only in the term ${n_i \choose x_i}$ which relates to the number of ways that you can order the $x_i$ successes in $n_i$ observations.

In that expression of the probability for separate observations, it is taken into account that each particular individual order is a different type of observation. In the expression for the grouped cases you take them all together and do not differentiate between different orders.

For example, if you have two successes out of four than this could have been each of the ${4 \choose 2 } = 6 $ different individual observations 1100 1010 1001 0110 0101 0011. And the probability for each of those individual cases differs with a factor $1/6$ from the probability for the grouped case.

Likelihood as invariant criteria

So, it does not matter if you do a comparison of models based on likelihood. For instance: likelihood ratio or AIC or BIC (or derived values like p-values, although the p-values are not always invariant for splitting and it depends on how you define 'extreme'). For a particular observation the term ${n_i \choose x_i}$ is just a constant factor (that only depends on the observation and not on the parameters $p_i$) that influences all the models equally.