1
$\begingroup$

I am trying to find a statistical difference between the success rate of several machine learning techniques using different activation functions. I obtained the success rates for each machine learning technique and the different activation functions. The following data is what I obtained:

+-------------+-----------------------+-----------------------+-----------------------+
|    Column   | Activation function A | Activation function B | Activation function C |
+-------------+-----------------------+-----------------------+-----------------------+
| Algorithm 1 | 90%                   | 70%                   | 50%                   |
| Algorithm 2 | 40%                   | 50%                   | 100%                  |
| Algorithm 3 | 60%                   | 90%                   | 90%                   |
+-------------+-----------------------+-----------------------+-----------------------+

As you can see there is one entry with a success rate of 100%, however that does not necessarily mean that algorithm 2 with activation function C is definitely the best approach. It might be the case that 1A, 2C, 3B and 3C are all good options since there might not be a significant difference among these algorithms. Which significance test should I use to test which algorithm(s) in combination with an activation function(s) is preferred over others?

The sample size is fairly small, per condition n=10.

$\endgroup$

1 Answer 1

1
$\begingroup$

As a simple start, convert your percentages to counts, then you get an contingency table and can use a chisquare test. I show below how to do this in R:

> x <- matrix(scan(), 9, 2, byrow=TRUE)
1: 9 1 4 6 6 4 7 3 5 5 9 1 5 5 10 0 9 1
19: 
Read 18 items
> x
      [,1] [,2]
 [1,]    9    1
 [2,]    4    6
 [3,]    6    4
 [4,]    7    3
 [5,]    5    5
 [6,]    9    1
 [7,]    5    5
 [8,]   10    0
 [9,]    9    1
> chisq.test(x)

    Pearson's Chi-squared test

data:  x
X-squared = 18.93, df = 8, p-value = 0.01524

Warning message:
In chisq.test(x) : Chi-squared approximation may be incorrect
> chisq.test(x, sim=TRUE, B=10000)

    Pearson's Chi-squared test with simulated p-value (based on 10000
    replicates)

data:  x
X-squared = 18.93, df = NA, p-value = 0.0148

So, yes, you can conclude that there are really some differences in success probabilities. To investigate that further you could try logistic regression.

$\endgroup$
4
  • $\begingroup$ There is some problem for logistic regression - there is zero in contingency table. I'm not sure there exists a good way to deal with it without resorting to bayesian approach. $\endgroup$ Apr 20, 2017 at 16:13
  • $\begingroup$ Why would the zero be a problem? $\endgroup$ Apr 20, 2017 at 18:18
  • $\begingroup$ Well, we wouldn't be able to get reasonable confidence interval for 'Activation function C' and 'Algorithm 2' and any comparisons with it would be meaningless. $\endgroup$ Apr 21, 2017 at 7:20
  • $\begingroup$ I really don't get how you would investigate that further with logistic regression. Can someone explain that? Got a similar problem here: stats.stackexchange.com/questions/575135/… $\endgroup$ May 13, 2022 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.