Estimate sample variance of return-on-ad-spend (ROAS)?

Question

I'm trying to make sense of the correct method to calculate the variance of ad-revenue for my marketing company. During an a/b test we need to estimate the sample variance to compute our confidence intervals for return-on-ad-spend (ROAS). Given the size of our datasets (millions to billions of users), these are the following metrics we have to use for estimation as we can't feasible calculate the sample variance directly:

• impressions (number of ads shown to users)
• installs (total number of installs regardless of if they make a purchase or not)
• converted_users (total number of unique purchasers, given they installed)
• ad_revenue (sum of all ad-revenue from converted_users)
• sum_squared_ad_revenue (sum of all squared ad-revenue; value computed in real time in database; e.g. $10 purchase = 100 sum squared)

The rate of installs (installs/impressions) is typically less than 0.02%, on average at my company, but the rate of purchasing given an install is typically around 15-20%. Furthermore, most converted_users spend very little (<$5) with a few large purchases (>100).

I know the correct way to estimate the variance would be to use the raw dataset where each row represents a user with a flag for if they made a purchase or not:

$$ s^2 = (1 / (n-1)) ∑(x_i - x_{mean})^2 $$

However, as mentioned our datasets are too large to realistically do that calculation repeatedly during an a/b test. Would the correct way to estimate the variance under this method be?:

$$ s^2 = (sum_{squared_ad_revenue} - (ad_{revenue})^2 / converted_{users}) / (converted_{users} - 1) $$

Given that most users never install, and of those that do, only a subset make a purchase, I wanted to verify my use of converted_users was correct and that I shouldn't use installs or impressions (for n).

Answer 1

You can easily compute the sample variance if you have an estimate of the mean $\bar x$ (do you?) and the sum of all squared revenue $\sum x_i^2$ (which you indicate you have).
Even if you can not compute the mean at the end, you can easily compute it in real time, by calculating a running sum (just as you do for the sum of squares). Then, an alternate formula for the sample variance is $(n-1)s^2=\sum x_i^2 - n {\bar x^2}$. You can see the same answer here as well on CV.