# Statistical significance between several percentages

I am trying to find a statistical difference between the success rate of several machine learning techniques using different activation functions. I obtained the success rates for each machine learning technique and the different activation functions. The following data is what I obtained:

+-------------+-----------------------+-----------------------+-----------------------+
|    Column   | Activation function A | Activation function B | Activation function C |
+-------------+-----------------------+-----------------------+-----------------------+
| Algorithm 1 | 90%                   | 70%                   | 50%                   |
| Algorithm 2 | 40%                   | 50%                   | 100%                  |
| Algorithm 3 | 60%                   | 90%                   | 90%                   |
+-------------+-----------------------+-----------------------+-----------------------+


As you can see there is one entry with a success rate of 100%, however that does not necessarily mean that algorithm 2 with activation function C is definitely the best approach. It might be the case that 1A, 2C, 3B and 3C are all good options since there might not be a significant difference among these algorithms. Which significance test should I use to test which algorithm(s) in combination with an activation function(s) is preferred over others?

The sample size is fairly small, per condition n=10.

As a simple start, convert your percentages to counts, then you get an contingency table and can use a chisquare test. I show below how to do this in R:

> x <- matrix(scan(), 9, 2, byrow=TRUE)
1: 9 1 4 6 6 4 7 3 5 5 9 1 5 5 10 0 9 1
19:
> x
[,1] [,2]
[1,]    9    1
[2,]    4    6
[3,]    6    4
[4,]    7    3
[5,]    5    5
[6,]    9    1
[7,]    5    5
[8,]   10    0
[9,]    9    1
> chisq.test(x)

Pearson's Chi-squared test

data:  x
X-squared = 18.93, df = 8, p-value = 0.01524

Warning message:
In chisq.test(x) : Chi-squared approximation may be incorrect
> chisq.test(x, sim=TRUE, B=10000)

Pearson's Chi-squared test with simulated p-value (based on 10000
replicates)

data:  x
X-squared = 18.93, df = NA, p-value = 0.0148


So, yes, you can conclude that there are really some differences in success probabilities. To investigate that further you could try logistic regression.

• There is some problem for logistic regression - there is zero in contingency table. I'm not sure there exists a good way to deal with it without resorting to bayesian approach. – Andrey Kolyadin Apr 20 '17 at 16:13
• Why would the zero be a problem? – kjetil b halvorsen Apr 20 '17 at 18:18
• Well, we wouldn't be able to get reasonable confidence interval for 'Activation function C' and 'Algorithm 2' and any comparisons with it would be meaningless. – Andrey Kolyadin Apr 21 '17 at 7:20