0
$\begingroup$

I'm currently working on a mobile app that offers premium subscription plans. I'm interested in testing whether increasing the price of the subscription might result in a significant increase in revenue even though the number of subscriptions might drop. I have data on the number of subscriptions and revenue for two random samples of the population base of users. The first sample (test group) has a new, higher price, while the second sample (control group) has the original price.

The data available is:

Control group:

  • Size: 4704
  • Number of subscriptions: 78
  • Revenue: 1377.22

Test group:

  • Size: 4683
  • Number of subscriptions: 65
  • Revenue: 1604.85

Edit: prices are constant in each group.

How to test that the increase in revenue is statistically significant?

$\endgroup$
5
  • $\begingroup$ Is the price constant within each group? $\endgroup$
    – dimitriy
    Feb 18, 2023 at 15:18
  • $\begingroup$ You will need to use a proportions or binomial test for something like this. $\endgroup$
    – Dave2e
    Feb 18, 2023 at 15:25
  • $\begingroup$ @dimitriy yes the prices are constant in each group. $\endgroup$ Feb 18, 2023 at 17:48
  • 1
    $\begingroup$ @Dave2e but the revenue is non-binomial $\endgroup$ Feb 18, 2023 at 17:50
  • 1
    $\begingroup$ That's okay if the change in revenue is a simple proportionate increase between the two groups. That would lead to a hypothesis of the form $H_0: r_0\mu_0 \ge r_1\mu_1$ where the $r_i$ are the relative per-subscriber revenues and the $\mu_i$ are the binomial expectations. That requires some subtlety to test properly, but permits use of the Binomial distribution. It becomes simple if you have so much information about the current mean $\mu_0$ that you can treat it as a known constant, because the hypothesis now concerns $\mu_1$ in a textbook fashion. $\endgroup$
    – whuber
    Feb 18, 2023 at 18:05

1 Answer 1

2
$\begingroup$

Since prices are the same within each group, the data you gave is a sufficient statistic for the analysis.

You can do this with regression or the traditional two-sample t-test:

. input ///
> treated sample_size tot_revenue 

       treated  sample_~e  tot_rev~e
  1. 0 4626 0
  2. 0 78   1377.22
  3. 1 4618 0
  4. 1 65   1604.85
  5. end

. generate avg_revenue = tot_revenue/sample_size

. list, clean noobs

    treated   sample~e   tot_re~e   avg_re~e  
          0       4626          0          0  
          0         78    1377.22   17.65667  
          1       4618          0          0  
          1         65    1604.85      24.69  

. /* Weighted Regression Version With Heteroskedastic Errors */
. regress avg_revenue i.treated [fw=sample_size], robust

Linear regression                               Number of obs     =      9,387
                                                F(1, 9385)        =       0.87
                                                Prob > F          =     0.3509
                                                R-squared         =     0.0001
                                                Root MSE          =     2.5907

------------------------------------------------------------------------------
             |               Robust
 avg_revenue | Coefficient  std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
   1.treated |   .0499207   .0535074     0.93   0.351    -.0549655    .1548068
       _cons |   .2927763   .0328779     8.90   0.000     .2283285    .3572242
------------------------------------------------------------------------------

. /* t-Test Equivalent */
. collapse (count) sample_size (mean) mean = avg_revenue (sd) sd = avg_revenue [fw=sample_size], by(treated)

. list, clean noobs

    treated   sample~e       mean         sd  
          0       4704   .2927763   2.254954  
          1       4683    .342697   2.888863  

. ttesti 4683 .342697 2.888863 4704 .2927763 2.254954, unequal

Two-sample t test with unequal variances
------------------------------------------------------------------------------
         |     Obs        Mean    Std. err.   Std. dev.   [95% conf. interval]
---------+--------------------------------------------------------------------
       x |   4,683     .342697    .0422148    2.888863    .2599361    .4254579
       y |   4,704    .2927763    .0328779    2.254954    .2283202    .3572324
---------+--------------------------------------------------------------------
Combined |   9,387    .3176808    .0267389    2.590643    .2652667    .3700949
---------+--------------------------------------------------------------------
    diff |            .0499207    .0535074               -.0549663    .1548077
------------------------------------------------------------------------------
    diff = mean(x) - mean(y)                                      t =   0.9330
H0: diff = 0                     Satterthwaite's degrees of freedom =  8844.82

    Ha: diff < 0                 Ha: diff != 0                 Ha: diff > 0
 Pr(T < t) = 0.8246         Pr(|T| > |t|) = 0.3509          Pr(T > t) = 0.1754

The estimated effect of seeing the higher subscription price is an additional .0499207 (or ~5 cents) of revenue per user (or 17%). This is the 1.treated coefficient in the regression output or the diff in the t-test output.

Unfortunately, that change is not statistically distinguishable from zero in this experiment since the p-value is 0.3509 (or half of that for the one-sided test, which makes sense here).

Having said that, you would need a test of about 67-85K users to detect a change that small, given how variable subscription revenue is.


Post Hoc Sample Size Calculation

. power twomeans .342697 .2927763, alpha(0.05) power(0.8) sd1(2.888863) sd2(2.254954) onesided

Performing iteration ...

Estimated sample sizes for a two-sample means test
Satterthwaite's t test assuming unequal variances
H0: m2 = m1  versus  Ha: m2 < m1

Study parameters:

        alpha =    0.0500
        power =    0.8000
        delta =   -0.0499
           m1 =    0.3427
           m2 =    0.2928
          sd1 =    2.8889
          sd2 =    2.2550

Estimated sample sizes:

            N =    66,640
  N per group =    33,320

. power twomeans .342697 .2927763, alpha(0.05) power(0.8) sd1(2.888863) sd2(2.254954)

Performing iteration ...

Estimated sample sizes for a two-sample means test
Satterthwaite's t test assuming unequal variances
H0: m2 = m1  versus  Ha: m2 != m1

Study parameters:

        alpha =    0.0500
        power =    0.8000
        delta =   -0.0499
           m1 =    0.3427
           m2 =    0.2928
          sd1 =    2.8889
          sd2 =    2.2550

Estimated sample sizes:

            N =    84,602
  N per group =    42,301
$\endgroup$
10
  • $\begingroup$ Thanks for the detailed answer. May I ask how did you reach the conclusion that 67-85k users are needed? Is that number per group or total of both groups. Thanks. $\endgroup$ Feb 18, 2023 at 20:37
  • $\begingroup$ I did a power calculation in my head for total sample size assuming 50-50 split. I will verify if I did the math right when I am near my computer again and add more detail. $\endgroup$
    – dimitriy
    Feb 18, 2023 at 20:46
  • $\begingroup$ @SinkingTitanic See above for the standard sample size output. $\endgroup$
    – dimitriy
    Feb 19, 2023 at 2:59
  • $\begingroup$ Can you please explain why you used [fw=sample_size] for the weights? If I understood correctly, that would assign weights to each observation as (1/sample size). In other words, observations with higher sample sizes will have less weight. Can you explain why did you chose that? Thanks a lot! $\endgroup$ Feb 20, 2023 at 12:21
  • $\begingroup$ Moreover, please correct me if I am wrong, but the "Two-sample t test with unequal variances" or sometimes called "Welch's t-test" assumes normality. How did you check for that given that we have only two observations per group? Thanks. $\endgroup$ Feb 20, 2023 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.