1
$\begingroup$

I am trying to find a statistical difference between the success rate of several machine learning techniques using different activation functions. I obtained the success rates for each machine learning technique and the different activation functions. The following data is what I obtained:

+-------------+-----------------------+-----------------------+-----------------------+
|    Column   | Activation function A | Activation function B | Activation function C |
+-------------+-----------------------+-----------------------+-----------------------+
| Algorithm 1 | 90%                   | 70%                   | 50%                   |
| Algorithm 2 | 40%                   | 50%                   | 100%                  |
| Algorithm 3 | 60%                   | 90%                   | 90%                   |
+-------------+-----------------------+-----------------------+-----------------------+

As you can see there is one entry with a success rate of 100%, however that does not necessarily mean that algorithm 2 with activation function C is definitely the best approach. It might be the case that 1A, 2C, 3B and 3C are all good options since there might not be a significant difference among these algorithms. Which significance test should I use to test which algorithm(s) in combination with an activation function(s) is preferred over others?

The sample size is fairly small, per condition n=10.

$\endgroup$
1
$\begingroup$

As a simple start, convert your percentages to counts, then you get an contingency table and can use a chisquare test. I show below how to do this in R:

> x <- matrix(scan(), 9, 2, byrow=TRUE)
1: 9 1 4 6 6 4 7 3 5 5 9 1 5 5 10 0 9 1
19: 
Read 18 items
> x
      [,1] [,2]
 [1,]    9    1
 [2,]    4    6
 [3,]    6    4
 [4,]    7    3
 [5,]    5    5
 [6,]    9    1
 [7,]    5    5
 [8,]   10    0
 [9,]    9    1
> chisq.test(x)

    Pearson's Chi-squared test

data:  x
X-squared = 18.93, df = 8, p-value = 0.01524

Warning message:
In chisq.test(x) : Chi-squared approximation may be incorrect
> chisq.test(x, sim=TRUE, B=10000)

    Pearson's Chi-squared test with simulated p-value (based on 10000
    replicates)

data:  x
X-squared = 18.93, df = NA, p-value = 0.0148

So, yes, you can conclude that there are really some differences in success probabilities. To investigate that further you could try logistic regression.

$\endgroup$
  • $\begingroup$ There is some problem for logistic regression - there is zero in contingency table. I'm not sure there exists a good way to deal with it without resorting to bayesian approach. $\endgroup$ – Andrey Kolyadin Apr 20 '17 at 16:13
  • $\begingroup$ Why would the zero be a problem? $\endgroup$ – kjetil b halvorsen Apr 20 '17 at 18:18
  • $\begingroup$ Well, we wouldn't be able to get reasonable confidence interval for 'Activation function C' and 'Algorithm 2' and any comparisons with it would be meaningless. $\endgroup$ – Andrey Kolyadin Apr 21 '17 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.