Unknown process outputs binary results, how to prove that this process is (or not) a Bernoulli trial

Question

I have an unknown process that produces binary results. I am trying to determine if this process is a Bernoulli trial.

From wikipedia:

In the theory of probability and statistics, a Bernoulli trial (or binomial trial) is a random experiment with exactly two possible outcomes, "success" and "failure", in which the probability of success is the same every time the experiment is conducted.

I do not know the actual success rate, if the success rate of each trial is equal, or if the trials are independent from each other.

However, I can collect data from the process in sequential trials with its corresponding successful trials.

Initial set of trials yielded 5 successes out of 999 trials.

• Success 1 was on trial 196
• Success 2 was on trial 388
• success 3 was on trial 593
• Success 4 was on trial 792
• Success 5 was on trial 999

What is the ideal way to demonstrate (perhaps with test statistics) if this process is or is not a Bernoulli trial. i.e.

• I already know it outputs binary results.
• does each trial have an equal or varying probability of success?
• are the trials independent?

---

Context:

A computer game uses an unspecified item drop system. Most item drop systems in computer games are Bernoulli trials with equal probability of success (i.e. an equal drop chance). However, I suspect that this particular system uses a count to determine a successful drop (e.g. success after ~200 trials). I want to design an experiment that will prove that the drop system uses an equal drop chance (Bernoulli trial) or a different system (e.g. a count based system).

---

My background:

I had only very basic statistics lesson in college many years ago. I am very fuzzy with test statistics. I have fair understanding of Bernoulli trials and binomial distributions.

---

My attempt:

Since I am trying to determine if the process is a Bernoulli trial with an equal chance of success p, if I can demonstrate that separate cumulative trials have different p, I can prove that the drop system does not use Bernoulli trial with an equal success chance.

To do this I collected 3 sets of data. Set A was a sequential 999 trials that resulted in 5 successes as stated above. Set B was 5 x 190 trials with a gap in between them. Set C was the gap between the 5 sets of 190 trials in B. i.e.

999 trials (A) - 190 (B) - 15 (C) - 190 (B) - 11 (C) - 190 (B) - 9 (C) - 190 (B) - 1 (C) - 190 (B) - 3 (C) - 190 (B)

This sampling yielded:

• A: 5 successes in 999 trials
• B: 0 successes in 950 trials
• C: 5 successes in 39 trials

The reason it is sampled this way is because I suspected that the drop counter is between 190 and 210, and that the drop counter resets after each successful trial. Every time I obtain a successful trial in Set C, I switch back to collecting Set B.

Using Clopper-Pearson interval (a.k.a. 'exact' method), α = 0.05, I calculated the interval of which Success chance p lies in, for each set of data.

• Set A: Success rate P_A: 0.00163 - 0.01164, 95% confidence
• Set B: Success rate P_B: 0.00003 - 0.00388, 95% confidence
• Set C: Success rate P_C: 0.04297 - 0.2743, 95% confidence

Since,

P_C ≠ P_A

P_C ≠ P_B

I conclude that the drop system observed is not a binomial experiment with equal success chance.

Is this an acceptable solution? If not, what is the correct method to approach this problem?

---

Research:

I've read through several posts related to this topic, however they are difficult for me to understand and I cannot be sure how to use those solutions on this particular question.

Test if two binomial distributions are statistically different from each other

Test if two binomial distributions comply with the same p

Is there a test for independence in a Bernoulli process?

Test on binomial distribution

Probablity distribution for different probabilities

estimate probability mass function from observed sample

Answer 1

As said in comments, any stricly binary (yes/no) variable is a Bernoulli variable, which is the same as an Binomial variable with parameter $n=1$ (and some $p$). There is no more possibilities, since such a variable has its distribution totally identified by the probability $p$ of occurence.

But when you have more than one bernoulli variable, and sum them up, to get the total number of occurences, there are other possibilities. Let there be $n$ such variables, with probability of occurence $p_i$ each. When all the probabilities $p_i$ are identical $p$, and the events are independent, the sum $X=X_1 + \dotso + X_n$ has the binomial distribution with parameters $n,p$. But with unequal probabilities and/or dependence, we get other possibilities.

First, the case with independence but enequal probabilities $p_i$, $$ \DeclareMathOperator{\E}{\mathbb{E}}\DeclareMathOperator{\V}{\mathbb{V}} \bar{p} = \frac1n \sum p_i. $$ Then the expectation $\E X$ still has the binomial value $np$ (by linearity of expectation), but the variance $\V X$ differs: $$ \V X = \V (\sum X_i) = \sum \V X_i = \sum p_i(1-p_i) \\ = \dots = n\bar{p}(1-\bar{p}) - \sum (p_i - \bar{p})^2 \\ < n \bar{p}(1-\bar{p}) $$ (after some algebra), so it is always smaller than the binomial variance!

There are of course many other possibilities, let us take the case of $n=2$, equal probabilities, but dependence:

Suppose that $$ P(X_1=0,X_2=0)= (1-p)^2 (1+\theta) \\ P(X_1=0,X_2=1)= p(1-p) (1-\psi) \\ P(X_1=1,X_2=0)= p(1-p) (1-\psi) \\ P(X_1=1,X_2=1)= p^2 (1+\theta) $$ (If you fix $p, \theta$, you can determine $\psi$. I leave that as an exercise ...) with a positive $\theta$, this is a kind of positive dependence, positive covariance. If I got my algebra right, the variance of $X=X_1 + X_2$ can be found to be: $$ \V X = 2 p (1-p) + \theta p^2 $$ so that, with a positive $\theta$, the variance is larger than binomial, but with a negative $\theta$, the variance is smaller than binomial!

Answer 2

Maybe I have a statistical test that might help you.

Let me first note that you want to know if your sequence of observations is a independent and identically distributed (i.i.d.) sequence of Bernoulli random variables. The problem is that a sequence can fail to be i.i.d. for a variety of reasons and there are a lot of randomness tests out there, each one better suited for a particular way in which a sequence fails to be i.i.d.. I believe that the number of runs test or the longest runs test (you can find its description in this book ) are two good candidates for you problem.

If none of the previous tests work let me propose a alternative approach. Let's suppose we have a sequence of i.i.d. Bernoulli r.v. $\{X_1,X_2,\ldots,X_n\}$, i.e., $X_i\sim Ber(p)$, for all $i$.

Now we will define the following random variables:

• $Y_1$ is the number of $0$ we shaw before the first $1$;
• $Y_2$ is the number of $0$ after the first $1$ that we shaw before the second $1$.
• In a similar way we define $Y_j$ for any value of $j\in\{1,2,\ldots\}$.

The interesting thing here is that (if we assume a i.i.d. sequence) all $Y_j$ have the same distribution which is a geometric distribution with success probability $p$, $geo(p)$:

$$\mathbb P (Y_j = k) = (1-p)^kp$$

Now we have our test. If your sequence is not a i.i.d. one, the distribution of the $Y_j$ you constructed from it would not be close to a $geo(p)$ distribution. This can be tested with any goodness of fit test. If you have no previous knowledge of the value of $p$ you can resort to estimating it from the sample.