Comparing families of classifiers on large datasets using mixed effect logistic regression models on individual questions

Question

I have a testing dataset of about 6000 images which I am going to try about 25 different neural networks on in a multi-class classification problem. Each network will belong to around 5 families (e.g. ResNet, EfficientNet, etc.) and within each family there will be several networks of increasing size (EfficientNetB0, B2 ... B7).

Each testing set case will belong to a class (it's a multiclass problem) and so the prediction can be correct or incorrect.

What would be the best stats approach to see if families differ in performance, accounting for the model size?

Rather than using % correct in a linear model (and only having n_networks results), I was thinking about treating each answer as correct/incorrect and using a multilevel logistic regression model, and as each network gets tested using the same testing cases, having the case as a random effect:

model <- glmer(correct_prediction ~ model_size + (1 | model_family/specific_model) + (1 | image_id), 
               data = data, family = binomial)

However I've not seen this approach before. Also, as it's a random effect, rather than a fixed effect I don't think it would allow me to compare individual families of networks?

Instead, could I do something like this:

model <- glmer(correct_prediction ~ model_family + model_size + (1 | model_family/specific_model) + (1 | image_id), 
               data = data, family = binomial)

Where I include model_family as a fixed effect as well as a random effect.

Finally, I'm almost certain model_size will have a non-linear relationship (small to medium models will offer big boosts, but medium to large less so), so I was thinking about modelling this using a restricted cubic spline, like so:

library(rms)
library(lme4)
library(lmerTest)
dat <- datadist(data)
options(datadist = 'dat')

model <- glmer(correct_prediction ~ rcs(model_size, 3) + model_family + (1 | model_family/specific_model) + (1 | image_id), 
               data = data, family = binomial)

Does this seem a sensible approach? As I've said, I've not really seen people using logistic regression for test scores, and yet to me it feels quite a logical (sorry) approach?

Answer 1

I am no kind of expert on this topic, but I would be very leery of dichotomizing the results into correct and incorrect when you have a more precise measure available. In particular, I'd be mindful of the quote from George Box (although he made it long before neural networks were available):

All models are wrong, but some are useful.

The question is how useful each model is, and that's not a yes/no question (except, perhaps, for some that are totally useless).