I have a data set of classification results that are binary (correct prediction or false prediction). Each entry stems from one of multiple models and contains a few more factors:

Model | Factor_1 | Factor_2 | Factor_3 | Classification_result |
Mod_1 | Green    | Car      | A        | TRUE                  |
Mod_1 | Green    | Car      | B        | TRUE                  |
Mod_2 | Green    | Car      | A        | FALSE                 |
Mod_2 | Green    | Car      | B        | FALSE                 |
Mod_3 | Green    | Car      | A        | TRUE                  |
Mod_3 | Green    | Car      | B        | TRUE                  |
Mod_1 | Black    | Plane    | C        | TRUE                  |
Mod_2 | Black    | Plane    | C        | TRUE                  |
Mod_3 | Black    | Plane    | C        | TRUE                  |

All models were applied to predict the same input data set. Thus, each combination exists for all models. However, in general, not all combinations between levels of the different factors exist. E.g., there might be no green planes and no black cars.

Additionally, the third factor might not be of interest and it would be useful to be able to control for its potential effect. (Dropping entries, e.g. until only one level of Factor_3 remains would be an option, but it would probably be better to use all the data. Another possibility would be to include Factor_3.)

In terms of the given example, I would like to test whether classification success is significantly different amongst

  1. models,
  2. levels of Factor_1,
  3. levels of Factor_2

and secondly, how the effect sizes amongst these three factors differ.

What is the method of choice in this case and why? (Maybe GLM plus some post-hoc test?)

In practice, would the following be an option? Why/why not?:

model <- glm(Classification_result ~ Model * Factor_1 * Factor_2, family=binomial(link="logit"), data=dataset)
emm <- emmeans(model, ~ Model * Factor_1 * Factor_2)
pairs <- pairs(emm, adjust = "tukey")
  • $\begingroup$ How do you get the classification result? $\endgroup$
    – Dave
    Jan 18 at 18:13
  • $\begingroup$ NB, between the models, the data are not independent. $\endgroup$ Jan 18 at 19:20
  • $\begingroup$ I'm guessing that the models are defined by which of these factors are included? So it seems to me you are trying to do model selection in an ad hoc way, rather than by using established methods. $\endgroup$
    – Russ Lenth
    Jan 19 at 1:51
  • $\begingroup$ The problem related to not having representation for all combinations of levels of factors is called "aliasing." Aliasing can cause a lot of prediction methods to blow up, often resulting in error messages that don't always straightforwardly tell you the model is aliased -- depending on the software used. As long as you can get a logistic regression coefficient for each k-1 level of the factors, you should be okay. But first use + in the model instead of *, since * are interaction terms that cause aliasing. Not sure R will explode on you if aliasing occurs, it may not. $\endgroup$
    – wjktrs
    Jan 19 at 5:06
  • $\begingroup$ @Dave, Russ Lenth The models are image classification models (deep learning). The models are not defined by the factors included. The factors are attributes of the data that may or may not have an impact on whether the respective image is classified correctly. E.g., say the classification is supposed to tell apart images of men and women, then the factors could be {"child", "adult", "elderly"} and {"European", "African", "Asian"} but we might lack images of elderly Europeans, for example. The research question would be whether the models work well across all those factors and where they fail. $\endgroup$ Jan 19 at 8:24


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.