2
$\begingroup$

Let's say that I have two samples:

A    B  C    D
100  5  17   12  #<- Method 1
90   2  4    15  #<- Method 2

And want to compare to test whether method 1 is different from method 2. However, that observations in the samples are tied together, say 100 and 90 come from the same market A. What is the proper way to test this?

I am pretty sure that lumping everything together and doing a t-test or Mann-Whitney U test is not the way to go.

$\endgroup$
12
  • $\begingroup$ What do "A", "B", "C", "D" mean? Could they correspond to factors that have a natural order? What do the numbers mean? (Are they counts, measurements of physical quantities, percents, ...?) How were the numbers obtained--are they the result of a complete census of something, of random samples, of samples of convenience, or something else? $\endgroup$
    – whuber
    Jul 2, 2014 at 16:54
  • $\begingroup$ We can think of "A", "B", "C", "D" as events; they have no order, but values corresponding to those events are paired. I am very sure that paired t-test (as suggested by @Greg Snow) is what I need. $\endgroup$
    – Akavall
    Jul 2, 2014 at 17:09
  • $\begingroup$ Exactly how would you apply the paired t-test to these data? It's still not possible to tell whether it would be appropriate or not because you haven't disclosed the (essential) information I mentioned. $\endgroup$
    – whuber
    Jul 2, 2014 at 17:58
  • $\begingroup$ @whuber Let's say we used marketing strategy method 1 in on half of customers in market A, and marketing strategy method 2 on the other half. Method 1 generated 100 purchases in market A, and method 2 generated 90 purchases in market A, and so on. Does this answer your question? $\endgroup$
    – Akavall
    Jul 2, 2014 at 18:32
  • $\begingroup$ Sort of--at least it shows me this is not a matter for a paired t-test! You have 100+90+5+...+15 total results and they are not paired. Although you could (somewhat artificially) pair the groups and perform a t-test of the differences 10,3,13,-3 against 0, it would have little power and cannot be recommended. (It reports that $p=0.21,$ which is not significant.) Many people would routinely begin with a chi-squared test to see whether any differences between the methods seem to hold. (There is a difference at the 5% level: $\chi^2=8.1$, $p\approx 0.04$ via simulation.) $\endgroup$
    – whuber
    Jul 2, 2014 at 18:48

2 Answers 2

1
$\begingroup$

If you are willing to assume normality of the differences (with your sample size, the assumption will be very important, but if your real data is much larger then the Central Limit Theorem makes this assumption less important) you can use a paired-T test, essentially take the difference between each pair (always the same direction) then do a 1 sample t test on the differences testing if the mean difference is 0.

There are non-parametric tests that will also test pairs and don't require the assumption of normality, but they are going to have very low power given your sample size (the sign test and exact permutation tests will have 0 power for any alpha level less than 0.0625).

$\endgroup$
0
1
$\begingroup$

Your observations appear to be counts (number of purchases). You would be best to use analyses that relate to that kind of data. A paired t-test won't be particularly suitable for a number of reasons (the variance of the difference isn't constant, for starters)

In this case, a chi-squared test of homogeneity will allow you to test for site-differences.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.