2
$\begingroup$

I need to perform a test to show that accuracy results from different models are statistically significant (or not). I have been looking into https://www.socscistatistics.com/tests/ but I do not know which test makes sense to perform.

I have multiple models with one accuracy score for each model. Which test should I perform for significance testing?

I have tried using the Single Sample Z Score Calculator, which gave me 3.7261 and T-Test Calculator for 2 Independent Means that returned 2769.93978, but I am not sure whether these tests are appropriate

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ You’re comparing two proportions, right? You know ways to do that ($\chi^2$ test, Fisher’s exact test, proportion z-test...). $\text{//}$ Why do you need to perform a hypothesis test to compare the two accuracy scores? $\text{//}$ Why use accuracy? Statisticians tend to prefer proper scoring rules like log loss. $\endgroup$
    – Dave
    Commented Dec 26, 2020 at 6:56
  • $\begingroup$ @Dave so if I would use Chi-Sq or Fisher's exact test, then my expected loss would be 0 for the two observations since it is an optimisation task with the objective to minimize the loss? $\endgroup$
    – Boris
    Commented Dec 26, 2020 at 7:14

1 Answer 1

Reset to default
0
$\begingroup$

A z test and similar tests are not appropriate since the data are paired. A better test would be McNemar's test, but this assumes both models predict on a single hold out set.

Even better would be to adopt a proper scoring rule and select models on that criterion. Frank Harrell writes about this in his book Regression Modelling Strategies if I recall correctly.

Improve this answer
$\endgroup$
3
  • $\begingroup$ seems that McNemar's test works on qualitative data only $\endgroup$
    – Boris
    Commented Dec 26, 2020 at 7:29
  • $\begingroup$ "Accuracy" is usually the word we use to describe the proportion of instances predicted correctly when the outcome is categorical. Is your question about regression and detecting differences in loss? $\endgroup$
    – Demetri Pananos
    Commented Dec 26, 2020 at 23:02
  • $\begingroup$ I have log loss and perplexity in my experiments $\endgroup$
    – Boris
    Commented Dec 27, 2020 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.