2
$\begingroup$

I have 30x 1-hour sessions where I assign one random person to each session to search for a specific type of content on a social media website. I record the number of pieces found in each session, resulting with 30 data points. From this, I can calculate my mean, but how do I calculate the confidence interval of that mean?

The individuals during the session can perform as many searches or look at as many pieces of content as they wish within that 1-hour session. We do not track the number of searches nor the number of pieces of content checked. We only record the content count found that meet our request.

My thought is that this is a Poisson process and we can assume a Poisson distribution. However, our mean doesn't always equal our variance. Here are my questions:

  • If Poisson distribution is OK to use, is it correct that my 95% confidence interval would be λ ±1.96*sqrt(λ/30) (30 being the number of sessions)?
  • What should I do if not all conditions are met for a Poisson distribution? Mean ≠ var.
  • Can we apply a standard normal distribution on this kind of problem?
  • Is 30 sessions is statistically significant.
$\endgroup$
4
  • $\begingroup$ a) About CI for Poisson λ: If you invert the approximate normal test for $n\lambda$ based on the total $T\sim \mathsf{Pois}(n\lambda)$ of $n$ observations, you get $T+2±1.96\sqrt{T+1},$ (conflating 1.96 with 2), which seems to come somewhat closer to 95% coverage than Wald-style $T±1.96\sqrt{T}.$ Somewhat analogous to Agresti CI for binomial $p.$ These can be converted to CIs for $\lambda$ by dividing enpoints by $n.$ (b) More generally: Have you considered dropping the Poisson assumption and using a bootstrap CI? $\endgroup$
    – BruceET
    Commented Oct 28, 2021 at 0:38
  • $\begingroup$ See This Q&A for coverage probabilities of the two styles of CI In Comment (a). $\endgroup$
    – BruceET
    Commented Oct 28, 2021 at 0:54
  • 3
    $\begingroup$ I guess you meant the mean and the variance are identical in Poisson distribution (not the standard deviation). $\endgroup$
    – Pitouille
    Commented Oct 28, 2021 at 11:25
  • $\begingroup$ yes. thank you. corrected. $\endgroup$
    – Cassova
    Commented Oct 29, 2021 at 6:28

1 Answer 1

6
$\begingroup$

Here are examples, using R, of the three kinds of confidence intervals mentioned in my comment (a), based on 30 observations from $\mathsf{Pois}(\lambda=50).$ [The nonparametric bootstrap CI uses the Poisson data, but without 'assuming' they are Poisson.]

set.seed(2021)
x = rpois(30, 50)
summary(x); sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  34.00   44.25   50.50   50.37   58.50   62.00 
[1] 8.405595  # sample SD; pop SD is 7.0711

a = mean(x);  a
[1] 50.36667
t = sum(x); t
[1] 1511

stripchart(x, meth="stack", pch=20)
 abline(v = a, col="red")

enter image description here

Wald and 'Agresti-style' 95% confidence intervals for $\lambda.$ [Start with intervals for $n\lambda,$ then divide by $n.]$

CI.1 = (t + qnorm(c(.025,.975))*sqrt(t))/30;  CI.1
[1] 47.82710 52.90623  # Wald CI

CI.2 = (t + 2 + qnorm(c(.025,.975))*sqrt(t+1))/30;  
CI.2
[1] 47.89293 52.97374  # CI from inverting test

d.re = replicate(2000, mean(sample(x,30,rep=TRUE))-mean(x))
UL = quantile(d.re, c(.975,.025))
mean(x) - UL
97.5%     2.5% 
47.46667 53.33333      # nonparametric bootstrap CI

Addendum: By contrast, at each iteration, a parametric bootstrap re-samples from a Poisson distribution with the observed mean 50.36667 from the original data as its mean. Thus the assumption is made that data are from a Poisson distribution.

set.seed(2021)
x = rpois(30, 50)
a = mean(x);  a
[1] 50.36667
d.re = replicate(4000, mean(rpois(30, a))-mean(x))
UL = quantile(d.re, c(.975,.025)) 
mean(x) - UL
   97.5%     2.5% 
47.73333 53.03333     # parametric bootstrap CI

Note: When used with Poisson data all four styles of CIs give about the same result. However, the nonparametric bootstrap CI is the only one of the four methods shown that would work for data that is not Poisson.

$\endgroup$
1
  • 2
    $\begingroup$ I love this. I hadn't thought of bootstrapping for this type of problem. Thanks! $\endgroup$
    – Cassova
    Commented Oct 28, 2021 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.