0
$\begingroup$

I'm confused about if t-test can only test means (sum / sample size), or if it can test sums as well (not normalizing for sample size).

Below is a passage from a trusted book.

Note that even though we want to increase overall revenue, we do not recommend using the sum of revenue itself, as it depends on the number of users in each variant. Even if the variants are allocated with equal traffic, the actual number of users may vary due to chance. We recommend that key metrics be normalized by the actual sample sizes, making revenue-per-user a good metric.

What test would test for differences in sum of revenue?

Edit: Data involved would be two series of unequal length. One observation would be a unique user placing a unique order.

For instance, control group has sales data in USD [12, 11, 9, 20] for a sum of 52 USD; and treatment group has sales data [19, 10, 8] for a total of 37 USD.

$\endgroup$
4
  • 2
    $\begingroup$ Please tell us what data you have. As the passage points out, a test of the sum would somehow have to incorporate a model of the variation in "actual number of users," which strongly implies a need to have many observations of the sums, not just one or two of them. $\endgroup$ – whuber Oct 1 '20 at 13:16
  • $\begingroup$ Unfortunately, the book didn't specify, but from what is practical to implement in industry, this will just be 2 observations of the sum - treatment group and control group. Multiple observations of the sum will require an order of magnitude larger sample size (and cost) which could not possibly justify using sums over per user metrics. $\endgroup$ – Peppershaker Oct 1 '20 at 18:33
  • 1
    $\begingroup$ Then you have little possibility of testing the sums from this information alone. $\endgroup$ – whuber Oct 1 '20 at 19:29
  • $\begingroup$ My apologies if I have misinterpreted your last comment, but the data involved would be two series of unequal length. For instance, control group has sales data in dollars [12, 11, 9, 20] for a sum of 52; and treatment group has sales data [19, 10, 8] for a total of 37. Thanks again $\endgroup$ – Peppershaker Oct 1 '20 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.