I am having some trouble confirming that I am on the right track and I hope someone can help me out.

I have a data set with 9 products each of which has received 8 treatments, including the control this makes a 9 Product x 9 Treatment matrix that was organised in a reduced Latin square.

Participants in the experiment were required to provide user ratings for each of the 9 products, one each with a different treatment applied. Consequently each participant seen each product once and each treatment once. This means that every participant contributed to each of the mean user ratings below.

I then worked out the mean user ratings for the control and each of the user ratings. Below is an example:

             Control  T1      T2      T3     T4     T5     T6      T7     T8
Mean Ratings 4.74     -1.77   -1.88   7.77   4.08   6.90   13.66   17.13  19.44

I want to find out the level of effect of each treatment in standard deviations from the mean of the control, which received no treatment.

  • The mean user ratings for the control and each of the treatments were taken from the same population and are selected at random
  • My mean user ratings range from -100 to +100
  • My data is normally or approximately normally distributed.
  • The sample size for each mean user ratings is > 30
  • The Standard Deviation is known.

Question: Is it possible using Z-tests to compare the the mean user rating of each of the treatments to the mean of the control? Is this the correct test?

So the comparisons will be in the from of Control V T1 then Control V T2 etc.

I did ask a similar question previously, but that focused on my efforts and what I have attempted so far. I wanted to ask this question separately to make sure I was on the right track before I keep going.

This has been driving me slightly crazy for a few days not so any help would be seriously appreciated, thanks.

  • $\begingroup$ "I want to find out the level of effect of each treatment in standard deviations from the mean of the control" Would it matter to you if treatment 1 had very high ratings for product 1 and very low rating for product 2, while treatment 2 made no difference to the ratings? Do you care about how a treatment affects individual products or do you only care about the performance of a treatment averaged out over all the products? $\endgroup$
    – Hugh
    Aug 14, 2016 at 16:44
  • $\begingroup$ For the moment I am only looking at the effect of the treatment averaged out over all of the products.I will be looking at the effect of the the treatment on specific products at a later stage, for this I plan on using T tests (I think). $\endgroup$
    – Deepend
    Aug 14, 2016 at 16:49

1 Answer 1


Following your comment saying that you are only interested in the performance of a treatment averaged out over all products; you are right to simplify the data as you did into the first table.

You might have considered using an ANOVA test but that would only tell you if there is a difference between any two groups out of the control or treatment. That means that it could pick up that treatment 1 performs differently from treatment 8, but you don't care if two individual treatments differ, only if treatments differ from the control.

Since you're comparing all the treatments with only one group-the control you should use a Z test, however you need to correct for multiple comparisons.

Comparing multiple things causes false-positives to occur more often. In a single Z test with $\alpha = 0.05$ the critical Z value is $\pm 1.96$. But you can imagine that if you had 100 treatments you are likely to find one of them with a Z score above $1.96$ occurring by chance.

Since all your treatments are sampled independently you can use the family wise error rate. If you want a significance of $\alpha$ then you will need to adjust this to $\alpha_{adjusted}=1-(1-\alpha)^8$ because you have 8 comparisons.

Compute a critical Z test statistic with the significance $\alpha_{adjusted}$. If any treatment has a Z score larger than this then it is significant.

  • $\begingroup$ Hey, thanks for that, its a really detailed answer and especially for adding the info regarding adjusting the family wise error rate. I have never heard of this before. $\endgroup$
    – Deepend
    Aug 14, 2016 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.