0
$\begingroup$

I've spent several hours online researching an appropriate means of using statistics to determine whether or not variation test results are meaningful, or just chance.

If the testing took the format of an email campaign, where two different sequences of emails were sent to a population of people, what would be an appropriate means of determining success?

As of and up to September 1st 2014, new email subscribers were sent 5 emails within 30 days. A new email campaign was conceived, where 5 emails were sent but with different content. From September 1st, as new subscribers signed up, each alternating address was placed into either the original sequence of emails or the new sequence.

The goal is that after 30 days the subscribers will become customers.

So I now have data that looks like this:

Group | Subscribers | Customers
A     | 500         | 50
B     | 500         | 60

So the conversion rate for group A is 10% and 12% for group B.

Group B, the new sequence, performs better than group A.

Is there an opportunity to use stats to underline this statement?

I was read lots of posts on Chi Square tests but the thing I couldn't reconcile was "expected". The web is full of examples of voting intentions of men Vs. women in terms of right Vs. left (Republicans and Democrats). In those examples the expected for Null hypothesis was 50/50.

My data is laid out in a way that looks like chi square examples but, since I have no "expected" I am unable to proceed.

Can anyone point me in the right direction here?

Improve this question
$\endgroup$
9
  • $\begingroup$ When you say "sent out to a population" ... do you mean he population about which you wish to make inference? Or do you simply mean they were sent out to some number in each group? If you intend the results to generalize outside those 1000 people (as you would if you wanted to say "B does better" about other people), they're not the target population but presumably some kind of sample. $\endgroup$
    – Glen_b
    Commented Oct 27, 2014 at 21:29
  • $\begingroup$ Assuming that these groups of 500 are intended to be random samples from the population of interest, you'd either use a two-sample proportions test (typically done as a z-test) or a 2x2 chi-square (A/B vs customer/not-customer). Properly organized the two should be equivalent. (There are other possible tests.).... there are many posts on this topic here. $\endgroup$
    – Glen_b
    Commented Oct 27, 2014 at 21:35
  • $\begingroup$ @Glen_b thanks for the feedback. RE your first comment I guess by population I mean it's not actually a sample: either you were placed in group A or B 50/50 so everyone is included, not just a sample. Presumably that removes inference from the equation since we have the whole population in question? Does this change things? Z-test still apply? The link you provided does not appear to work $\endgroup$
    – Doug Fir
    Commented Oct 27, 2014 at 22:06
  • $\begingroup$ An example of a two-sample proportions test is [here](stats.stackexchange.com/questions/78474/correct-test-for-proportions/); there are several links describing it here. There's a chi-squared test here but for more than two groups, that shows the subtraction of the count of interest from the population exposed to obtain the second set of counts for the chi-squared test...(ctd) $\endgroup$
    – Glen_b
    Commented Oct 27, 2014 at 22:07
  • $\begingroup$ (ctd) ...Both tests are mentioned in the table [here](en.wikipedia.org/wiki/Statistical_hypothesis_testing#Common_test_statistics). $\quad$ Um, I didn't provide a link in my first two comments. It's not clear to me that you do have the whole target population. What use is it to know that something has a higher proportion on some fixed group from the past? What was the purpose of this analysis? $\endgroup$
    – Glen_b
    Commented Oct 27, 2014 at 22:09

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.