4
$\begingroup$

I have some process that succeeds or fails with probability $p$. I can do distinct simulations to estimate $p$:

  1. Run $N$ simulations of a single process, record $N$ samples of a $\text{Bernoulli}(p)$, do GLM with binomial family to estimate $\log(p/(1-p))$.
  2. Run $N$ simulations in which I continue running the process until a success, record $N$ samples of a $\text{Geometric}(p)$, do GLM with gamma family to estimate $p$.

What is the theoretical difference? Which way is better? I thought (1) is nicer but then I can't get $p$, only $\text{logit}(p)$, right? (see answer below).

UPDATE: I do regression because there is a simulation parameter t and I want to determine if p depends on t and how. Also, the above Ns are not the same, because each simulation in the geometric design is roughly equivalent to 1/p simulations in the Bernoulli design. But this is part of the question - is it better to make each trial on its own (Bernoulli) or to make sequences of trials? How does this design affect the power/significance/other statistical measures of both estimation of p and estimation of the regression parameters?

$\endgroup$
2
$\begingroup$

There is not much difference between the methods other than that the number of trials in method 1 is more predictable than in method 2. Consider the following example:

Say we tried 5 simulations using method 2 and the results are:

2, 2, 1, 4, 1

So our estimate of p = 1/mean = .5

This also means that we have run 10 trials:

0, 1
0, 1
1
0, 0, 0, 1
1

We can see this as a method 1 simulation and see 5 successes out of 10 trials, which corresponds to a estimate of p of .5

This is no coincidense. For method 2 our estimate of $p$ is 1/mean, which is $\frac{N} {\sum_i k_i}$ (whereby $k_i$ are the number of trials till success). We also know that this represents a method 1 simulation with $\sum_i k_i$ trials and $N$ successes, so our estimate will again be $\frac{N} {\sum_i k_i}$. So there is no real difference between method 1 and method 2 other than that the cost of method 1 is more predictable than the cost of method 2. As long as you compare like with like, that is, compare a method 2 simulation with a method 1 simulation with $\sum_i k_i$ trials, you will find no difference in the results with respect to the point estimates. As you use different variance functions with gamma and logit families you might get slight differences in standard errors. In Stata I would inspect this like so:

clear all
set obs 100
gen t = _n <= 50
gen p = .1 + .3*t
gen u1 = runiform() < p
local i = 1
sum u1, meanonly
local stop = r(mean) == 1

while `stop' == 0 {
    local j = `i'
    local i = `i' + 1
    gen u`i' = runiform() < p if u`j' == 0
    sum u`i', meanonly
    local stop = r(mean) == 1
}

egen k = rownonmiss(u*)

glm k t, family(gamma) link(power -1)
margins, over(t) predict(xb)

gen id = _n
reshape long u, i(id) j(iter)
glm u t, family(binomial) link(logit)
margins, over(t)

The relevant outputs are:

for the gamma model:

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
           t |
          0  |   .0988142   .0112884     8.75   0.000     .0766895     .120939
          1  |    .462963    .052888     8.75   0.000     .3593043    .5666216
------------------------------------------------------------------------------

for the logit model:

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
           t |
          0  |   .0988142   .0132661     7.45   0.000     .0728132    .1248152
          1  |    .462963   .0479803     9.65   0.000     .3689232    .5570027
------------------------------------------------------------------------------

So the predicted $p$ are exactly the same and there are slight differences in the standard error.

$\endgroup$
  • $\begingroup$ Great, thanks. I'll emphasize: running a single process (failure/success) has a cost, say x time on average. For Bernoulli with N1 simulations the cost will be N1*x. For geometric with N2 simulations, in which I run 1/p trials on average because I run until success, it will be N2*x/p. But this just says that for a total CPU time X, I can run either N1=X/x Bernoulli simulations or N2=X*p/x geometric simulations. $\endgroup$ – yoavram Jul 9 '13 at 12:30
  • $\begingroup$ Another comment is that I use GLM because there is a simulation parameter t and I want to see how p depends on t. I've updated the question following this comment. $\endgroup$ – yoavram Jul 9 '13 at 12:31
  • $\begingroup$ Yep, thanks, I agree that the Bernoulli approach makes the number of trials more predictable. I was wondering if there are other differences regarding the estimation error and the regression error. $\endgroup$ – yoavram Jul 9 '13 at 13:26
  • 2
    $\begingroup$ @yoavram For any given number of trials I believe they contain the same information about the parameter. $\endgroup$ – Glen_b -Reinstate Monica Jul 9 '13 at 23:22
  • $\begingroup$ So it doesn't matter that for Bernoulli I use logistic regression (GLM with logit link) and for geometric I use GLM with 1/x link (Gamma family)? $\endgroup$ – yoavram Jul 10 '13 at 9:23
1
$\begingroup$

I thought (1) is nicer but then I can't get p, only logit(p), right?

Not so.

$\hat p = \frac{1}{1+\exp(-\hat\eta)}$ where $\eta = \text{logit}(p)$

That is, you just transform back.

$\endgroup$
  • $\begingroup$ Yeah, I just figured out the expit function will get me p en.wikipedia.org/wiki/Logit. I'll edit the question. But still, which design is better? $\endgroup$ – yoavram Jul 9 '13 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.