Compare two distributions with bias

Question

I have been programming in R and have a dataset containing the results (succes or not) of two Machine Learning algorithms which have been tried out using different amounts of parameters. An example is provided below:

type success paramater_amount
a1     0       15639
a1     0       18623
a1     1       19875
a2     1       12513
a2     1       10256
a2     0       12548

I now want to compare both algorithms to see which one has the best overall performance. But there is a catch. It is known that the higher the parameter_amount, the higher the chances for success. When checking out the parameter amounts both algorithms were tested on, one can also notice that a1 has been tested with higher parameter amounts than a2 was. This would make simply counting the amount of successes of both algorithms unfair.

What would be a good approach to handle this scenario?

Answer 1

EDIT: I changed the answer from a simple decision rule to a more statistical answer.

You would be well advised to use a weights-based approach to handle the problem. One simple way to do so is to multiply the success count of type $a1$ by $\frac{t_2}{t_1}$, where $t_1$ and $t_2$ are the associated totals of the parameter_amount variable: $t_1=54137$ and $t_2=35317$.

So, if the original contingency table was $$\begin{array}{c|c|c|c} & \text{Successes} & \text{Failures} & \\ \hline \text{Type a1} & a & b & a+b\\ \hline \text{Type a2} & c & d & c+d\\ \hline \text{} & a+c & b+d & a+b+c+d\\ \hline \end{array},$$ it should be replaced by

$$\begin{array}{c|c|c|c} & \text{Successes} & \text{Failures} & \\ \hline \text{Type a1} & \frac{t_2}{t_1} a & b & \frac{t_2}{t_1}a+b\\ \hline \text{Type a2} & c & d & c+d\\ \hline \text{} & \frac{t_2}{t_1}a+c & b+d & \frac{t_2}{t_1}a+b+c+d\\ \hline \end{array}$$

This way, one penalises type $a1$ for its unfair advantage by changing only one of the four non-marginal entries in the contingency table.

Now, it is appropriate to use a one-tailed Fisher's exact test because the individual observations are independent.

Let $p_1,p_2$ be the success rates (the underlying theoretical ones for which we are conducting inference).

Let $\alpha$ be our chosen confidence level. A $(1-\alpha)\times100\%$ confidence interval for $p_1-p_2$ is given by

$$ \hat{p_1}-\hat{p_2} \pm z_{1-\frac{\alpha}{2}} \sqrt{\frac{\hat{p_1}(1-\hat{p_1})}{n_1}+\frac{\hat{p_2}(1-\hat{p_2})}{n_2}},$$

where the $\hat{p_i}'s$ and the $n_i's$ are the weight-corrected proportions (success rates) and sample sizes, and $z_{1-\frac{\alpha}{2}}$ denotes the corresponding percentile point of the standard normal distribution.

I hope this helps. Let me know if anything is unclear.