Is a Z-Test, the right test to compare the mean of several sub groups to a control mean?

Question

I am having some trouble confirming that I am on the right track and I hope someone can help me out.

I have a data set with 9 products each of which has received 8 treatments, including the control this makes a 9 Product x 9 Treatment matrix that was organised in a reduced Latin square.

Participants in the experiment were required to provide user ratings for each of the 9 products, one each with a different treatment applied. Consequently each participant seen each product once and each treatment once. This means that every participant contributed to each of the mean user ratings below.

I then worked out the mean user ratings for the control and each of the user ratings. Below is an example:

             Control  T1      T2      T3     T4     T5     T6      T7     T8
Mean Ratings 4.74     -1.77   -1.88   7.77   4.08   6.90   13.66   17.13  19.44

I want to find out the level of effect of each treatment in standard deviations from the mean of the control, which received no treatment.

• The mean user ratings for the control and each of the treatments were taken from the same population and are selected at random
• My mean user ratings range from -100 to +100
• My data is normally or approximately normally distributed.
• The sample size for each mean user ratings is > 30
• The Standard Deviation is known.

Question: Is it possible using Z-tests to compare the the mean user rating of each of the treatments to the mean of the control? Is this the correct test?

So the comparisons will be in the from of Control V T1 then Control V T2 etc.

I did ask a similar question previously, but that focused on my efforts and what I have attempted so far. I wanted to ask this question separately to make sure I was on the right track before I keep going.

This has been driving me slightly crazy for a few days not so any help would be seriously appreciated, thanks.

Answer 1

Following your comment saying that you are only interested in the performance of a treatment averaged out over all products; you are right to simplify the data as you did into the first table.

You might have considered using an ANOVA test but that would only tell you if there is a difference between any two groups out of the control or treatment. That means that it could pick up that treatment 1 performs differently from treatment 8, but you don't care if two individual treatments differ, only if treatments differ from the control.

Since you're comparing all the treatments with only one group-the control you should use a Z test, however you need to correct for multiple comparisons.

Comparing multiple things causes false-positives to occur more often. In a single Z test with $\alpha = 0.05$ the critical Z value is $\pm 1.96$. But you can imagine that if you had 100 treatments you are likely to find one of them with a Z score above $1.96$ occurring by chance.

Since all your treatments are sampled independently you can use the family wise error rate. If you want a significance of $\alpha$ then you will need to adjust this to $\alpha_{adjusted}=1-(1-\alpha)^8$ because you have 8 comparisons.

Compute a critical Z test statistic with the significance $\alpha_{adjusted}$. If any treatment has a Z score larger than this then it is significant.