3
$\begingroup$

I am trying to measure a statistically significant lift in sales and want to confirm appropriate methodology

Test Design

  1. Created "Matched Markets" of 20 stores (10 in control vs. 10 in test)

Source Data (represented as total sales during test period):

Control Test

 100   185
 125   180
 135   170
 130   172
 140   190
 170   177
 165   165
 160   197
 162   185
 134   196

Methodology

I am running a one-tailed t-test since I do not know the SD of the population and we are hypothesizing that Test is greater than control. Based on this, I am calculating a p-value of .00003 with the result is significant at p < .05.

I want to confirm a t-test is appropriate here and I am not violating any assumptions around populations (since the stores in test vs. control are in different states) or anything else.

I also wanted to confirm whether you are ALSO able to declare significance since the average of the test group (181.7) is greater than the Confidence Interval of control at 95% (calculated to be (142.1 ± 14)

Improve this question
$\endgroup$
3
  • $\begingroup$ Are you running a paired t-test? If so, are your stores the same for each row? $\endgroup$
    – Reid
    Commented May 11, 2017 at 19:12
  • 1
    $\begingroup$ Hi Reid! I guess that is a good question - is independent or paired more appropriate? The test vs. control stores were matched to be a close as possible in terms of sales, demographic, etc. That being said, would they be dependent because they're both the same stores just in different markets? $\endgroup$
    – user3682157
    Commented May 11, 2017 at 19:26
  • $\begingroup$ I think there's a bit to untangle here, so please see my response below! $\endgroup$
    – Reid
    Commented May 11, 2017 at 19:30

1 Answer 1

Reset to default
3
$\begingroup$

Overview

Your instinct to use a t-test is ok, but maybe a non-parametric equivalent would be better. More importantly, your data and experimental design might not match the conclusions you want to draw.

Paired vs Independent

From what you've said, it seems like the stores are independent, but I'm not sure if you've run a paired or independent t-test. If you have 20 stores that you measured once, then you'll need to run an independent sample t-test.

Assumptions for a t-test

It looks like your data are continuous, but your sample size is too low to really conclude that your data are normally distributed. To play it safe, I'd advocate the use of a non-parametric version of a t-test.

Non-parametric t-tests

For an independent t-test equivalent, check out the Mann-Whitney U test, which is available in R, python, SPSS and JASP. This is the most conservative analysis you could do under these circumstances. When I ran your numbers, I found a p<0.001, which looks good!

Two Tailed vs One Tailed

Although you could theoretically run a one-tailed t-test, it's generally thought that these are not conservative enough (and in fact that p-values themselves don't do enough to eliminate alpha error). My advice is to use a two-tailed assumption. My p-value above was using a two-tailed assumption, so you'll be just fine.

Interpretation

From what you've presented, it's hard to conclude that there was a statistically significant "lift in sales". You've said that the stores are in separate states (which is why you needed to run an independent t-test), but how were the stores doing before you started?

Instead of total sales, you should take the difference between sales during the 'test' period and subtract the baseline from them. Then run an independent t-test on the "difference" between baseline and test periods.

Improve this answer
$\endgroup$
8
  • $\begingroup$ One last question for you - in regards to your last point around test - baseline, that sounds like a pre vs. post test? If this is the case, why would that be independent test? Wouldn't that be a paired t-test since you are sampling the same stores twice? $\endgroup$
    – user3682157
    Commented May 11, 2017 at 19:42
  • $\begingroup$ Also, wouldn't a chi-square be a good choice for a pre vs. post since you know the expected values (the pre test) vs. the actual values (post test)? $\endgroup$
    – user3682157
    Commented May 11, 2017 at 19:44
  • 1
    $\begingroup$ The paired t-test would tell you average change over time and whether it was significant. You'd have no direct way of inferring whether it was driven by the control or the test group. Subtracting baseline from testing period would control for that, then you'd analyze the difference between all 20 stores. That way you can see whether the ones who got the 'test' improved more, because your outcome variable is literally the improvement. $\endgroup$
    – Reid
    Commented May 11, 2017 at 19:47
  • 1
    $\begingroup$ Please don't run a chi-square test. That's for frequency data and not applicable here. $\endgroup$
    – Reid
    Commented May 11, 2017 at 19:47
  • 1
    $\begingroup$ +1, especially for the Interpretation. Alternatively, one could look not at additive uplifts per store in the test vs. baseline period, but at multiplicative uplifts. These might be more comparable between larger and smaller stores. Right now, every single Test store has sales at least as high as the corresponding Control store, so if a pattern like this existed on uplifts, I wouldn't even use a statistical test, just an eyeball test. (Or even run a simple binomial test. What's the probability of ten sales increases in a row? $0.5^{10}<0.001$.) $\endgroup$
    – Stephan Kolassa
    Commented May 11, 2017 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.