0
$\begingroup$

I have three models against a single dataset. I want to see which model is better. Is repeated measures ANOVA the appropriate test, or regular one-way ANOVA?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Neither of these are. One-way and RM ANOVA are for comparing means, not models. $\endgroup$
    – Noah
    Commented Sep 4, 2023 at 20:28
  • $\begingroup$ You use one-way ANOVA when you have 3 or more groups/conditions to compare. Depending on whether the scores of the 3 conditions were produced by the same participants at different time interval will allow you to perform a repeated measured ANOVA. if the scores were produced by independent groups then it is a regular One-way ANOVA. $\endgroup$
    – Adekunle Morakinyo
    Commented Sep 4, 2023 at 22:03

1 Answer 1

Reset to default
0
$\begingroup$

Which method you use to determine which is "better" depends on what you mean by "better" and that, in turn, depends on the type of model (is it linear regression? Logistic? Something else?) Are the models nested?

For OLS regression, you could compare $R^2$ (plain or adjusted); AIC, BIC, and their variants; log likelihood; or maybe something else.

Me? I would look at graphs of how well each did compared to the others, then try to decide whether I thought one was better for the purposes I had. For instance, in some cases, being wrong by a huge amount on a few observations is worse than being wrong by a small amount on a lot of observations; sometimes it's reversed.

I'd look at plots of the residuals of each model against the others; maybe a Tukey mean difference plot (aka Bland Altman plot). Maybe QQ plots.

Also, I'd bear in mind that sometimes a small effect size is very important. If theory says it should be big, for instance. That might mean that a model that is worse on various statistical tests is better for your purposes.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I'm using binary classification models. To compare these models, I'm not sure if the groups would be independent or not using cross-validation (CV). Hence not sure if repeated measures applies here - thoughts? The measure by the way I plan to use is AUC and the number of variables used per model per CV run. $\endgroup$
    – user19226726
    Commented Sep 9, 2023 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.