For example, say you have a black box that has a number of n-sided dice in it. You have 4-sided dice, 6-sided, 8, 10, 12, 20 and so on. The die sides are all A except for one side that says B, e.g.
You select one of these dice in the black box. You aren't sure which one. You start rolling it and the output you see from the black box is either A or B. What is the minimum number of times would you need to roll the die to determine which which n-sided die you have?
My initial thoughts would be to track the probability of B coming up as you make more and more rolls and look at the coefficient of variation. Once the coefficient of variation had dropped below some threshold, the probability of rolling B would have stabilized. Then you relate that probability to the number of sides the die has (16.6% = 6 sided die). The number of rolls required to reach this stabilization would then be the minimum number of rolls.
Alternatively, my other thought was to look at standard error and confidence intervals. Make a number of rolls and keep track of the probability of rolling a B. Once the confidence interval for the mean probability of a B has dropped below whatever threshold you have (e.g. +/- 1%) that would be your criterion for saying the probability had stabilized and reached its 'true' value, and you then get the minimum number of rolls.
Both of these approach I think would require multiple runs. You could have have a succession of B rolls (or a longer string of A rolls) that would then throw of the coefficient of variation / standard error and make it take longer to drop below whatever threshold you decide on.
Is there another / better way of approaching this? It seems to be similar to a coin toss problem, like how many tosses you would need to determine if a coin is fair or not, but I haven’t been able to follow / apply that to my problem when the 'true' probability is unknown (.e. not 50%).