1
$\begingroup$

I want to test if the outcomes of different algorithms are different with statistical significance. I am testing 4 algorithms, each of which output 0 or 1 after every run.

I am running these algorithms for multiple series of multiple runs, e.g. 2 series of 5 runs in this example. So, the output of each algorithms looks like this (using Python):

alg1 = array([[0, 0, 1, 1, 1], [1, 1, 0, 1, 0]])
alg2 = array([[1, 1, 0, 0, 0], [1, 1, 1, 1, 1]]) 
alg3 = array([[0, 0, 1, 1, 1], [1, 0, 1, 0, 0]])
alg4 = array([[1, 0, 0, 1, 1], [1, 1, 0, 1, 1]])

My idea was to use the chi-squared test to see if the outcomes are different. Again, using Python:

obs = np.array([alg1, alg2, alg3, alg4])
scipy.stats.chi2_contingency(obs)

Is the chi-squared test the correct test to see if the outcomes of these algorithms are independent?

Improve this question
$\endgroup$
2
  • $\begingroup$ Is there any reason to believe that there will be serial correlations within the runs (eg, the 1st run will be more like the 2nd than the 4th), or that runs within a series will be more similar that runs in a different series? $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 12, 2017 at 15:35
  • 1
    $\begingroup$ @gung Runs within a series will be more similar than runs in a different series. For every serie, a new dataset is created uniformly at random. While for every run within a serie, a random sample from the dataset is used. $\endgroup$
    – JNevens
    Commented Apr 12, 2017 at 15:42

1 Answer 1

Reset to default
0
$\begingroup$

Your setup doesn't quite match the idea of independence. Instead, you want to know if the estimated proportions are the same. You could rearrange your data to make that work with a chi-squared test (although I think there is value in the conceptual clarity of referring to it this way), but the true population parameters in each series are likely to differ slightly. That means there should be more variability from one series to the next than you would otherwise suspect. A way to deal with these issues is to treat each series as a single binomial realized value and then fit a logistic regression model using the quasibinomial to account for the overdispersion. There are other ways, but I think that would be simplest.

I don't know how this can be done in Python, but I can demonstrate it in R (and it should be possible to call R from Python): 

alg1 = rbind(c(0, 0, 1, 1, 1), c(1, 1, 0, 1, 0))
alg2 = rbind(c(1, 1, 0, 0, 0), c(1, 1, 1, 1, 1)) 
alg3 = rbind(c(0, 0, 1, 1, 1), c(1, 0, 1, 0, 0))
alg4 = rbind(c(1, 0, 0, 1, 1), c(1, 1, 0, 1, 1))
l = list(alg1, alg2, alg3, alg4)
d = data.frame(alg=as.factor(rep(1:4, each=2)), success=as.vector(sapply(l, rowSums)))
d$failure = 5-d$success
d
#   alg success failure
# 1   1       3       2
# 2   1       3       2
# 3   2       2       3
# 4   2       5       0
# 5   3       3       2
# 6   3       2       3
# 7   4       3       2
# 8   4       4       1
m = glm(cbind(success, failure)~alg, d, family=quasibinomial)
summary(m)
# ...
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)
# (Intercept)   0.4055     0.7333   0.553    0.610
# alg2          0.4418     1.0734   0.412    0.702
# alg3         -0.4055     1.0266  -0.395    0.713
# alg4          0.4418     1.0734   0.412    0.702
# 
# (Dispersion parameter for quasibinomial family taken to be 1.290496)
# 
#     Null deviance: 7.5403  on 7  degrees of freedom
# Residual deviance: 6.3730  on 4  degrees of freedom
anova(m, test="LRT")
#      Df Deviance Resid. Df Resid. Dev Pr(>Chi)
# NULL                     7     7.5403         
# alg   3   1.1673         4     6.3730   0.8243
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.