I have a model with logistic (binomial) likelihood, with number of successes and failures as a response variable. I am comparing various models, which can be of different granularity. Different granularity means that the binomial observations can be either:
- grouped together (successes and failures summed up) for each site, or
- evaluated separately for each visit (there can be multiple visits to each site).
So, I am looking for model quality criteria, which wouldn't change with the site/visit granularity; i.e. which would produce the same result regardless of how the binomial observations are grouped.
I developed bunch of model comparison criteria, but as you can see below, apart from the AUC, all of them change with granularity. Here below is the evaluation of a single model using different criteria - first column shows the site-level granularity, second column the visit-level granularity:
per_site per_visit
AUC_1h 0.97175420 0.97175420
AUC_1h_weighted 0.97033082 0.97033082
R2_avgScore 0.49352020 0.42906301
R2_dev 0.68408469 0.53648654
R2_LR 0.62293855 0.53648654
R2_dev
is pseudo $R^2$ based on deviance, R2_LR
is based on likelihood, McFadden’s - see definitions here.
The problem with binomial likelihood:
$$\prod_{i}{n_i \choose x_i}p_i^x(1-p_i)^{n_i-x_i}$$
is that it contains the binomial coefficient ${n_i \choose x_i}$, which is the (only) term which depends on the granularity.
Since I don't want to stick just to AUC, I tried to look for other pseudo-R-squared methods for one which would be granularity-invariant. The Cox & Snell did look promissing:
because the binomial coefficients would cancel each other out in the fraction. However, there are two problems with this:
It needs a modification: $N$ needs to be set up so that it is granularity invariant. So instead of putting $N$ as number of records, one would put $N$ as the total sum of all successes and failures (which doesn't change with granularity). Would that make sense? Or is there any conceptual problem with this modification?
The maximum of this criteria is not one, which makes it difficult to interpret. This is addressed by Nagelkerke / Cragg & Uhler’s pseudo R-squared:
but here again, the denominator will ruin the granularity-independence again, as it depends on the binomial coefficient.
So how to address this?
- Is there a way to reasonably modify Cox & Snell? (See the 2 points above)
- Or would it make sense to just use all of these likelihood-based criteria, and just calculate the likelihood without the binomial coefficients? Would that make sense?
- Is there another reasonable, granularity-invariant criteria?
- Is my way of thinking alright, or is it conceptually broken (for example because the granularity is so important, that it doesn't make sense to look for granularity invariant criteria)? Why?