0
$\begingroup$

I am pulling meta data fields from a population of 100M documents. These meta fields include things like Author Name, Author Location, Keywords, etc. Some documents have all of these meta fields, some documents have a few of these fields, and some documents don't have any. We have no idea what the meta field coverage rates are for the population, but would like a best guess from a sample before we process the entire set ( eg. 45% of documents will include an Author Name ... 5% will include an Author Location ).

Basic sample size calculation suggest that I need to analyze 166 random documents to be 99% confident that my sample represents the population ( 99% Confidence Level ), while having a Confidence Interval of plus or minus 10% on the coverage rates we discover.

Intuitively, however, this does not seem like the correct method to me. If a meta field occurs very rarely in the full population, it seems like I'd have to adjust my population or change the Confidence Interval for that specific Meta Field.

How should I design this test to get valid coverage rates for meta fields representative of the complete population of documents?

$\endgroup$
0
$\begingroup$

Unfortunately, as you notice, the length of a CI for the parameter of a Bernoulli random variable depends on the same variable. At 50% the CI is shorter, at the tails it is longer. So you cannot give a uniform error bound for all fields, unless in all fields the same proportion of entries is empty.

$\endgroup$
  • $\begingroup$ So say I only care about meta fields that occur in my sample population >25% of the time. How do I calculate the required sample size to reach a given Confidence Level and Interval at 25% occurrence? Would this represent "worst case" confidence for fields with higher occurrence percentages? $\endgroup$ – T. Brian Jones Dec 17 '14 at 20:42
  • $\begingroup$ Yes, confidence intervals will only become shorter as $p$ increases towards 50%. By the way, (stats.stackexchange.com/questions/51851/…) covers your question entirely, with references. $\endgroup$ – Horst Grünbusch Dec 17 '14 at 22:04
  • $\begingroup$ Horst, do you have 'shorter' and 'longer' flipped around there? The variance $\pi(1-\pi)/n$ has a maximum at 0.5, which implies the standard error is largest when $\pi=0.5$, and so - at least for large $n$ - CIs should also be larger for $\pi=0.5$ than for small $\pi$. It's possible I've misunderstood your point, but either way it would be good to make it clear. $\endgroup$ – Glen_b -Reinstate Monica Dec 18 '14 at 0:56
  • $\begingroup$ Right, I confused something as I thought about it in Pearson-Clopper-bounds. So the worst case length can be found for $\pi = 0.5$. $\endgroup$ – Horst Grünbusch Dec 18 '14 at 8:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.