I have a data source with an accuracy of 69.3% (when compared with ground truth data). When a model is applied with 3 different parameters the resulting accuracies are 73.088, 74.912 and 72.89. How can I say that the model resulted in significant accuracy improvement over the single data?

  • 1
    $\begingroup$ Essentially same question as here. Same clarifications needed here as there. Please edit Question. If this is a class assignment, please use self-study tag. $\endgroup$ – BruceET Jun 3 at 4:51
  • $\begingroup$ Of the two nearly-duplicate questions, I am answering this one (a) because you asked first and (b) because you correctly proposed (via topic tag) use of a chi-squared test. // However, please edit your question to show the true number of 'subjects' for each of the four percentages you give in your Question. Then if you have follow-up questions one of us can answer in terms of the actual data. $\endgroup$ – BruceET Jun 3 at 5:40

This cannot be answered without knowing the actual counts. Hypothetically, if you had 693, 731, 749, and 720 successes (under four different respective conditions) out of 1000 trials. then you would have the contingency table TBL below. [Please understand that 1000 is just a guess to be able to illustrate how to run the test in R. You need to use the actual counts for a proper analysis of your data.]

x = c(693, 731, 749, 720)
y = 1000 - x
TBL = rbind(x, y);  TBL
  [,1] [,2] [,3] [,4]
x  693  731  749  720
y  307  269  251  280

Then a chi-squared test of homogeneity rejects the null hypothesis that successes are uniformly distributed across the four categories; rejection is at the 5% level.

chisq.test(TBL, cor=F)

        Pearson's Chi-squared test

data:  TBL
X-squared = 8.2372, df = 3, p-value = 0.04136

Looking at a table of the Pearson residuals, it seems the the greatest relative discrepancies are between column 1 and columns 2 and 3.

chisq.test(TBL, cor=F)$resi
       [,1]       [,2]       [,3]       [,4]
x -1.124815  0.2881758  0.9574873 -0.1208479
y  1.818367 -0.4658626 -1.5478662  0.1953618

Even though the main hypothesis is barely significant at the 5% level, ad hoc tests comparing columns 1 & 2 and 1 & 3 might be appropriate without undue risk of false discovery. (Especially, so if true counts are much larger than my guess of 1000.)

Comparison of 1 & 3 is significant at the 1% level:

chisq.test(TBL[,c(1,3)], cor=F)

        Pearson's Chi-squared test

data:  TBL[, c(1, 3)]
X-squared = 7.7948, df = 1, p-value = 0.00524
  • $\begingroup$ Let me rephrase the question If a system/machine runs an accuracy of 69.3%. I have introduced an improvement and the system now runs on accuracy of 73.088, 74.912 and 72.89. So I want to see if the improvement was significantly effective. In that case, we have three sample size. Wouldn't our reference value will always be 69.3% to compare with the new three values? $\endgroup$ – MSilvy Jun 3 at 15:33
  • $\begingroup$ Suppose you take 69.3 as a standard. Then by a process you don't understand or won't reveal you somehow get three other values you believe to be comparable and all three are above 69,3. You have no basis to claim the three observations are normally distributed. So you'd have to use a nonparametric test (as suggested in the parallel post). Problem is three is too few observations for a nonparametric test to find significance at 5% level. P-value of one-sided test couldn't be below $1/8.$ // Let me say again percentages without knowing base count are essentially worthless. Glad this is closed. $\endgroup$ – BruceET Jun 3 at 19:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.