Equivalent success rate I'm training models whose objective is to guess the correct image out of $n_i$ images.
Let's say that in settings where $n_i = 2$ the model manages to guess correctly 90% of the time.
In settings with $n_i = 4$ the model scores much lower, which is not surprising, since the chance level for $n_i = 4$ is lower ($p = 0.25$) than in the former setting ($p = 0.5$).
In order to be able to compare the performances in these two settings, I need to find what success rate is "equivalent" to those 90% under $n = 4$.
Is there a name for what I'm trying to achieve?
Is there a convenient function in scipy or similar package?
My idea was to:

*

*convert the success rates to error rates ($q = 1 - p$) to minimize floating point errors,


*calculate the probability mass $x$ on a binomial distribution of $n_i = 2$,


*calculate the percent point function at point $x$ on a binomial distribution of $n_i = 4$,
My python/scipy code for the above is
def equivalent_error_rate(q1, err, q2, n=500):
    cdf = st.binom.cdf(err * n, n, q1)
    return st.binom.ppf(cdf, n, q2) / n

The problem is that this approach is very prone to floating point errors and in the scenario above I get the same result for error rates $< 0.001$.
 A: Comment: To get started.
You don't say the number of images you use to assess a subject; whether your criterion for success is to do better than by chance alone $(1/2$ for 2 choices, $1/4$ for 4 choices); or whether you are testing a null hypothesis at the 5% level of significance.
So it's difficult to know how you want to make the 2-choice and 4-choice scenarios equivalent.
Here is one attempt to do that. Let the number of trials be 50 and take rejection of the null hypothesis of no better than average at the 5% level as evidence the subject is doing well. If that's a reasonable interpretation of you question and my assumptions make sense, then what I show below may be helpful.
If not, please edit your question to be more specific about the number of pictures to be identified (trials) and your criterion for a successful performance. Then one of us can try again.
Two-choice scenario. You can say that the subject is doing better than by guessing by testing $H_0: p \le 1/2$ against $H_a: p > 1/2.$ In that case, a test of $H_0$ is rejected at around
the 5% level of significance if the subject gets 31 or more right out of 50.
Under $H_0,$ the number $X$ guessed correct out of $n = 50$ has
$X \sim \mathsf{Binom}(n=50,p=1/2).$ And
$$P(X \ge 31\, |\,H_0) = 1 - P(X \le 30\, |\, H_0) = 0.059.$$ If you insist on
a significance level below 5%, then insist on $X \ge 32$ in order to reject $H_0.$
[Binomial distributions are discrete, so you can't usually get 5% exactly.
In R statistical software pbinom denotes a Binomial CDF; qnorma is the inverse CDF (quantile function).]
qbinom(.95, 50, .5)
[1] 31
1- pbinom(30, 50, .5)
[1] 0.05946023
1- pbinom(31, 50, .5)
[1] 0.03245432


R code for figure above:
x = 0:50;  PDF = dbinom(x, 50, .5)
plot(x, PDF, type="h", col="blue", main="BINOM(50,.5)")
 abline(v=30.5, col="red", lwd=2, lty="dotted")
 abline(h=0, col="green2")

Four-choice scenario.
You can say that the subject is doing better than by guessing by testing $H_0: p \le 1/4$ against $H_a: p > 1/4.$ In that case, a test at around
the 5% level of significance rejects $H_0$ if the subject gets 18 or more right out of 50.
Under $H_0,$ the number $X$ guessed correct out of $n = 50$ has
$X \sim \mathsf{Binom}(n=50,p=1/4).$ And
$$P(X \ge 18\, |\,H_0) = 1 - P(X \le 17\, |\, H_0) = 0.055.$$ If you insist on
a significance level below 5%, then insist on $X \ge 19$ in order to reject $H_0.
qbinom(.95, 50, .25)
[1] 18
1 - pbinom(17, 50, .25)
[1] 0.05512336
1 - pbinom(18, 50, .25)
[1] 0.02873316


