# What would be the most appropriate significance test for my scenario?

I have three groups of users $-$ $G1$, $G2$ and $Control$. The users in each of these three groups are different but are carefully selected and have similar properties. I treat each group with a unique treatment. $G1$ is treated by $T1$, $G2$ is treated by $T2$ and the $Control$ group is not treated at all. For each of these groups, I measure a variable $X$ before the treatment and after the treatment. I have the following questions -

[1] I would like to measure if my treatment (both $T1$ and $T2$) made a difference. In this case, I need to compare the variable $X$ for users in groups $G1 + G2$ to $Control$. Since I do not have any knowledge of the underlying distribution of data, I plan to use a Wilcoxon Signed Rank Test. Is this correct? If not, which other test would be appropriate?

[2] I want to measure which of the two treatments ($T1$ or $T2$) performed better. How do I do this? Should I perform a statistical test between users in $G1$ and $G2$ OR perform a statistical test between $G1$ and $Control$ AND $G2$ and $Control$ and interpret from the corresponding result?

For the first problem, you can start with an ANOVA/regression and check the distributional assumptions (which are not about the data but about error as estimated by the residuals). If they are violoated you can then do a nonparametric test or robust test.

For the second problem, since your goal is to compare T1 to T2, you should compare G1 to G2. You can try a t-test to start.

Or you can do both within the ANOVA/Regression model using different contrasts.

• For 1, Why is ANOVA a better choice? Pardon me but I am a beginner to this kind of statistics. For 2, Is it appropriate to compare G1 and G2 with a statistical significance test? Nov 15, 2012 at 11:39
• If the assumptions of ANOVA are met, it is slightly more powerful; it is also more familiar to most scientific audiences, which doesn't make it statistically better but is nevertheless useful. Also, the parameter estimates are easy to interpret; the W statistic is not intuitively meaningful. Nov 15, 2012 at 12:02
• The only issue is that the ANOVA requires data to be normally distributed. I can not afford to make that assumption. Nov 15, 2012 at 12:11
• @PeterFlom: you note that the distributional assumptions are not about the error as estimated by the residuals. I am confused. I always assumed that we needed independence of observations, normality and equality of variances in each group - and the last two are properties of the residuals. What did I misunterstand? Nov 15, 2012 at 13:35
• @StephanKolassa That was a mistake, I have fixed it. The assumptions are not about the data they are about the error. Nov 15, 2012 at 14:05

I went for Fisher's Exact Test and Chi-Square Contingency Test to answer my question. It looked more elegant and intuitive [0,1].

[0] http://en.wikipedia.org/wiki/Fisher's_exact_test