0
$\begingroup$

Does the unit of measurement change how we calculate sample size?

For example, if we were to run an A/B test on ads with our target metric being click-through-rate, and we are doing a sample size calculation:

We would maybe use some kind of proportions z-test.

Does it matter if we are calculating a sample size for ad impressions, users, or something else as our "unit of measurement"?

Improve this question
$\endgroup$
9
  • $\begingroup$ Sorry I don't get it. CTR = #clicks/ #impressions, what exactly is the "unit of measurement" thing here? $\endgroup$
    – lpounng
    Mar 10 at 4:40
  • $\begingroup$ @lpounng CTR should not be the number of impressions, it should be number of users who see the variants. Users can see the variant multiple times $\endgroup$
    – Demetri Pananos
    Mar 10 at 4:41
  • $\begingroup$ no you are wrong. Directly from Google Ad's definition: CTR is the number of clicks that your ad receives divided by the number of times your ad is shown: clicks ÷ impressions = CTR. For example, if you had 5 clicks and 100 impressions, then your CTR would be 5%. $\endgroup$
    – lpounng
    Mar 10 at 4:44
  • $\begingroup$ You can of course define your own metric, just don't call it CTR. $\endgroup$
    – lpounng
    Mar 10 at 4:45
  • $\begingroup$ @lpounng See my answer for why "impressions" is probably not the best denominator. Moreover, I don't really care what google Ad's says. $\endgroup$
    – Demetri Pananos
    Mar 10 at 4:46

1 Answer 1

Reset to default
2
$\begingroup$

Yes.

The "unit of measure" roughly determines the denominator and hence the precision in the estimate of the mean (here, the click through rate).

Impressions vs users randomized is a great example. Imagine you randomized users to two arms for an experiment. One user in one fo the groups just LOVES clicking your ad.

If your "unit of measure" was the impression, you would have to count each click from this user as an affirmative outcome.

If your "unit of measure" was the individual user, then you see the user clicked at least once and you're done. (Actually, there is some more nuance here. You would either want to measure the outcome of the user's first time seeing the variant if possible or else a) measure the time to the outcome, if it happens, using a model which can account for censoring, or b) define a period of time under which the outcome can possibly occur. That is just some flavour though, not really important to what we are discussing).

Clearly, the expected outcome and its variance depend on the unit. So yes, it matters, and it matters quite a lot.

Improve this answer
$\endgroup$
8
  • $\begingroup$ thank you for the response. i think it makes sense... but i dont quite get how to change my power analysis and sample size calculation approach? all of these sample size calculators require the same inputs like MDE, baseline conversion rate, power, confidence level, etc - so how do i modify my calculation based on whether or not we are measuring impact on a user or impression? $\endgroup$
    – jc315
    Mar 12 at 9:02
  • $\begingroup$ any thoughts on this Demetri? your response was very helpful and i think along the lines of what is being implied in my ask, just not sure how it directly impacts how i approach sample size calculation $\endgroup$
    – jc315
    Mar 14 at 17:14
  • $\begingroup$ You need to determine the click rate per users. This estimate will then be used to calculate the effect size used in the sample size calculation $\endgroup$
    – Demetri Pananos
    Mar 14 at 17:20
  • $\begingroup$ so is the main difference then 1) how we calculate the baseline conversion rate? i.e. click rate on a user level vs an impression level? and 2) when we evaluate the results of the experiment, we would calculate results on an user vs impression level? does anything else change in our approach (i.e. different formulas or something). to add more complexity to this - our CTR rate here is handled by a 3rd party, which is aggregated at a campaign_level - and so we do not have this data on a user or impression level. is this a major issue and impact how we can calculate sample size / run an experimnt $\endgroup$
    – jc315
    Mar 14 at 17:51
  • $\begingroup$ 1) Yes. 2) Yes. If you don't have data at the user level then i think the conversation is moot. Not much you can do about that $\endgroup$
    – Demetri Pananos
    Mar 14 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.