3
$\begingroup$

Context:

I am collecting Twitter data through a source that allows me to take a statistical sample of tweets that match a filter. I want to determine the appropriate sample rate in order to achieve a certain statistical confidence in my estimate of the actual number of tweets that would match that filter.

Sampler details:

For each tweet, the sampler calculates a new random floating point number uniformly distributed in the range [0,100], then compares that number to a given threshold (e.g. 1.6 for 1.6%). If the number is less than the threshold, the tweet is sampled; otherwise, it is discarded.

An example:

Let's say @justinbieber gets 500,150 mentions per day. I filter first to get all tweets that mention @justinbieber, then I sample at a probability of 1%. That should give me something like 5,001, which I'd then divide by 1% to get an estimate of 500,100 mentions/day.

There's two sources of error here:

1) Rounding error. I am necessarily reducing the precision of my count data, then extrapolating back to the original.

2) Sampling error. While the sampler uses uniformly distributed probability in the range, I may not get exactly the percentage of tweets I am requesting.

The Goal

I want to calculate the appropriate sample probability I should use in order to achieve an estimate of twitter mention count with a certain confidence. For example, I might want to say with 95% confidence that my estimate is within 2% of the actual count. Given both sources of error here, what formulas or techniques allow me to calculate the proper probability to use in my sampler given an estimated per-day count?

Notes

The best solution should work equally well with small counts (<40) and large counts. It's certainly acceptable to sample at 100% for twitter handles with small counts, but there is a fixed cost per-tweet, and thus minimizing the number of tweets required is important when possible.

$\endgroup$

1 Answer 1

2
$\begingroup$

There's some ambiguity. Suppose they separate all the Justin Bieber tweets (JBT), and then they give you exactly 10%. Then you divide that value (50k) by the sample rate (10%), get 500k, the original number, with NO statistical error.

If the filter is applied to the tweets, and then they evaluate whether they are JBT (which is what is implied by your question) then you have a simple form of a Horvitz–Thompson estimator. The reason for mentioning that is that the HT estimators can handle various probabilities; that info can come in handy if you change the sampling rate over time (for budget purposes, for example).

On page 27 of http://www.math.umt.edu/patterson/549/Horvitz-Thompson.pdf there is a formula for the variance of the total (i.e. around the 500,000 JBT).

Let's look at that formula: enter image description here

In the first term (to the left of the +) we know pi=.1 (probability of selection 10%). Since y=1 if it's a JBT and 0 otherwise, the fact that we don't know v (the sample including both JBT and non-JBT) doesn't matter. Further, the second term (to the right of the +) is the covariance term, but but with a 10% sample this can be ignored. This gives a standard error for the population estimate of 500,000 of 2,225.

We can check the reasonableness of this estimate by assuming various number of total Tweets on any topic (from say, 5e6 to 5e10) and seeing what we get from a simple binomial. This varies from 2,121 to 2,236 -- i.e. it doesn't vary much at all. (which is good because we don't know that number).

Working this through to a sampling rate for your requested confidence interval gives a sampling rate of 1.9%

$\endgroup$
2
  • $\begingroup$ Thanks very much for your answer. Sorry if I was unclear in the question -- I can arrange it either way - i.e. taking 10% of tweets, then matching for JBT; or matching for JBT, then taking 10% of tweets. They both have the same computational and per-tweet costs. Will this formula work for both cases? Also, for some twitter handles, there may only a small handful of mentions per day. Does this formula work well in the case of small numbers (<40 for example)? $\endgroup$
    – Alec
    Mar 19, 2012 at 16:22
  • $\begingroup$ @Alec: Should work for small numbers as well. If you can do it either way, then I'd match for JBT and then take 10%, because you will really only have rounding error in the total estimation (e.g. 10% of 100 to 109 total tweets will be 10). At very high sampling rates the FPCF (finite population correction factor) may enter in, which will lower your variance. $\endgroup$
    – zbicyclist
    Mar 19, 2012 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.