I have been programming in R and have a dataset containing the results (succes or not) of two Machine Learning algorithms which have been tried out using different amounts of parameters. An example is provided below:

type success paramater_amount
a1     0       15639
a1     0       18623
a1     1       19875
a2     1       12513
a2     1       10256
a2     0       12548

I now want to compare both algorithms to see which one has the best overall performance. But there is a catch. It is known that the higher the parameter_amount, the higher the chances for success. When checking out the parameter amounts both algorithms were tested on, one can also notice that a1 has been tested with higher parameter amounts than a2 was. This would make simply counting the amount of successes of both algorithms unfair.

What would be a good approach to handle this scenario?


EDIT: I changed the answer from a simple decision rule to a more statistical answer.

You would be well advised to use a weights-based approach to handle the problem. One simple way to do so is to multiply the success count of type $a1$ by $\frac{t_2}{t_1}$, where $t_1$ and $t_2$ are the associated totals of the parameter_amount variable: $t_1=54137$ and $t_2=35317$.

So, if the original contingency table was $$\begin{array}{c|c|c|c} & \text{Successes} & \text{Failures} & \\ \hline \text{Type a1} & a & b & a+b\\ \hline \text{Type a2} & c & d & c+d\\ \hline \text{} & a+c & b+d & a+b+c+d\\ \hline \end{array},$$ it should be replaced by

$$\begin{array}{c|c|c|c} & \text{Successes} & \text{Failures} & \\ \hline \text{Type a1} & \frac{t_2}{t_1} a & b & \frac{t_2}{t_1}a+b\\ \hline \text{Type a2} & c & d & c+d\\ \hline \text{} & \frac{t_2}{t_1}a+c & b+d & \frac{t_2}{t_1}a+b+c+d\\ \hline \end{array}$$

This way, one penalises type $a1$ for its unfair advantage by changing only one of the four non-marginal entries in the contingency table.

Now, it is appropriate to use a one-tailed Fisher's exact test because the individual observations are independent.

Let $p_1,p_2$ be the success rates (the underlying theoretical ones for which we are conducting inference).

Let $\alpha$ be our chosen confidence level. A $(1-\alpha)\times100\%$ confidence interval for $p_1-p_2$ is given by

$$ \hat{p_1}-\hat{p_2} \pm z_{1-\frac{\alpha}{2}} \sqrt{\frac{\hat{p_1}(1-\hat{p_1})}{n_1}+\frac{\hat{p_2}(1-\hat{p_2})}{n_2}},$$

where the $\hat{p_i}'s$ and the $n_i's$ are the weight-corrected proportions (success rates) and sample sizes, and $z_{1-\frac{\alpha}{2}}$ denotes the corresponding percentile point of the standard normal distribution.

I hope this helps. Let me know if anything is unclear.

  • $\begingroup$ This method intuitively makes a lot of sense, thanks! I did the calculations for my real dataset and I got two numbers 64 and 46 but I'm wondering what conclusions I can now derrive from it. Shouldn't I still run some kind of statistical test which can lead me to some conclusion with a certain confidence interval? I'm thinking of maybe making a 2 x 2 matrix with the weighted successes/failures and using using a Fisher's test as I only have 100 sample size to compare them? $\endgroup$ – Astarno Dec 15 '19 at 17:52
  • $\begingroup$ Thanks for your ellaboration. To me, only having to correct one of the four entities sounds a bit counter-intuitive, won't it ruin the original ratio between successes/failures? I've tried translating your answer to R code to be sure I'm interpreting it correctly (look my original question), is this the correct way to do it? $\endgroup$ – Astarno Dec 16 '19 at 14:16
  • $\begingroup$ Whilst the approach alters the ratio, it is sound if your primary purpose is to compare the two algorithms fairly. Indeed, you would like to be able to say that $p_1=0.4$ is worse than $p_2=0.4$, for example, taking into account parameter_amount. $\endgroup$ – Mickybo Yakari Dec 16 '19 at 15:16
  • $\begingroup$ Are you familiar with the difference between one-tailed and two-tailed Fisher's test ? Given your question, you want to perform a one-sided Fisher's test with H1: $p_1>p_2$ or H1:$p_1<p_2$. Don't forget to add alternative = "one.sided" in the fisher.test function. Feel free to let me know if anything is unclear. Do you know domain-knowledge that guarantees $p_1 \geq p_2 $ or $p_1 \leq p_2 $? $\endgroup$ – Mickybo Yakari Dec 16 '19 at 15:17
  • $\begingroup$ Yes, I remember those concepts from an earlier statistics course I had. I currently use alternative="greater" which returns a p-value of 0.8547. I'm interpreting that as that there is not enough evidence to reject the null hypothesis in which alogorithm 1 performs equally to algorithm 2. On the oppositive however, we also can't definitively conclude that algorithm 1 does perform better than algorithm 2. Is this the correct interpretation? $\endgroup$ – Astarno Dec 16 '19 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.