I am doing a study to evaluate the effect of two dietary supplements on body composition.

Literature suggests that Treatment 1 (Dietary Supplement 1) affects body composition with some side effects on a few blood profiles. We would like to know if Treatment 2 (Dietary Supplement 2) has the same effect (or even better) on body composition than Treatment 1, while not having those adverse effects on blood profiles.

Here is the dataset summary:

  • Total Number of Subjects: 171

  • Total Number of Groups: 3

    • Control, 56 Subjects

    • Treatment 1, 58 Subjects

    • Treatment 2, 57 Subjects

  • There are 12 repeated measures (each done 3 times), so 36 columns in total. The measures include Body Weight, Body Mass Index, Body Fat Mass, and bunch of blood profiles like HDL, LDL, etc. Hence, some measures are related (Body weight and BMI) while others aren't. It seems (see below) that some measures follow Gaussian distribution and some do not.

  • All these measurements were done 3 times during the study.

Assuming A is body weight. The body weight is measured at three time points during the study. Therefore, I have three body weight measures A1, A2, and A3 for all groups (Control, Treatment 1, Treatment 2). Thus, the sample data-set looks like (each row represents a subject):

Thus, the sample data-set looks like

Here are the problems:

  1. For within-group test, I've found that I should use repeated measures ANOVA with Bonferroni correction as post-hoc test for Gaussian measures and Friedman's test with Wilcoxon signed-rank (for each pair) as post-hoc test for non-Gaussian measures. Are these correct choices (particularly the post-hoc ones)?

  2. For between-group tests, should I use one-way ANOVA with Bonferroni correction as post-hoc test for Gaussian, and Kruskal-Wallis with Dunn's test as post-hoc for non-Gaussian measures?

  3. For this data-set, what does the output of post-hoc analysis of between-group effects mean exactly? For example, if B3 is significantly different between groups, does the post-hoc show which group's B3 is significantly different from which other group?

  4. If I want to report all within- and between-groups effects for each measurement along with post-hoc results, that would be a lot! Any advise on reporting analyses on such data-sets?

Normality tests

I used Shapiro-Wilk test in R to test the normality of each measure for each group. Here are histograms for B1 in two groups (as an example of non-normal data):

enter image description here

enter image description here


  • We are not 100% sure that Treatment 1 is effective (there is some controversy in the literature about how effective it is), so we included the Control group to see whether Treatment 1 is even effective, then compare it to Treatment 2.

  • About the time points, there are three of them (at the beginning, after 6 weeks, and 12 weeks). This is to see how fast or slow are the effects of both treatments, compared to each other and also compared to placebo.

  • While body composition measurements sound depended, it is not always the case. One could loose weight without her BFM changing at all, or it could be solely because of change in BFM.

  • $\begingroup$ @Glen_b I used SPSS and R for running this test, and all they needed were just the vectors (e.g. A1, A2, A3). I also used the group variable as a factor. $\endgroup$
    – NULL
    Commented Nov 14, 2015 at 1:54
  • $\begingroup$ @Glen_b Here is a page I followed for running Shapiro-Wilk test in SPSS: statistics.laerd.com/spss-tutorials/… $\endgroup$
    – NULL
    Commented Nov 14, 2015 at 1:55
  • $\begingroup$ @Glen_b I ran the test in R based on information in this page: stat.ethz.ch/R-manual/R-devel/library/stats/html/… $\endgroup$
    – NULL
    Commented Nov 14, 2015 at 1:56
  • $\begingroup$ Thanks, I think I get it now. Imagine there were no problem. In your preferred analysis would you need to assume a normal distribution across subjects? (Even if you do, is it at very sensitive to that assumption?) $\endgroup$
    – Glen_b
    Commented Nov 14, 2015 at 4:49
  • 1
    $\begingroup$ They assume a normal distribution of the error term, which would be checked for reasonableness by examining the residuals. [Note that testing isn't very useful -- failing to reject doesn't mean you have normality and rejecting doesn't necessarily imply the analysis will not perform well.] -- it may be that you have non-normality (in fact, real data are never drawn from normal distributions), but I haven't seen anything that would lead me to be concerned about it. $\endgroup$
    – Glen_b
    Commented Nov 14, 2015 at 6:53

2 Answers 2


I think one could write a whole book dealing exclusively with your question (and I am definitely not qualified to write it). So without any attempt at providing a comprehensive answer, here are some points that can hopefully be helpful.

Confirmatory vs. exploratory approach to analysis

As you note yourself, you have a very rich dataset and you can test a lot of things. We can quickly compute the number of meaningful tests: you have $12$ measures; each was measured $3$ times in $3$ groups. So if we count all pairwise tests, it will be $3$ tests per group and $3$ tests per measurement time, i.e. $18$ tests per measure, i.e. $216$ tests. You are obviously aware of the lurking multiple comparisons problem (remember the green beans comic?), but if you are normally happy to use $\alpha=0.05$ and were to use e.g. Bonferroni adjustment then you would have to use $\alpha = 0.05/216\approx 0.002$ and to risk not finding any significant effects because you do not have enough power.

This is of course not a unique, but in fact a very common situation.

Broadly speaking, you can adopt one of the two approaches.

Confirmatory approach insists on strict adherence to the rules of significance testing. You should formulate your one or several (but as few as possible) research hypotheses in advance and carefully plan which statistical tests you are going to carry out. To mitigate the multiple comparisons / low power problem, you should try to design your tests such that you use as few tests as possible while having maximal power to detect what you really want to detect. For example, you might want to combine your measures into some composite or pooled measures that are likely to be most affected by Treatment 1 or 2. Or you can pool over measurement times. Etc. In any case, you try to boil down all your data to a couple of crucial comparisons, and then you do only those, applying Bonferroni (or similar) adjustment. It's important that all of that is planned before you have ever looked at the data (because after looking at the data you will be tempted to change your tests).

Alas, in practice, this is often hardly possible.

Exploratory approach, in contrast, is like biting the bullet: you have a lot of rich data, so why not explore all sorts of relationships that are present in there. You will do lots of comparisons and lots of tests, you will adjust your analysis strategy depending on what you see in the data, but whatever -- this is all exploratory. You cannot do that if you are doing a clinical trial, but in more basic research this if often the only way to go. All $p$-values that you get out of this approach should be taken with a (big) grain of salt, though. In fact, some would say that you should not run or report any significance tests at all, but usually tests are still done. There is a good argument not to use multiple comparisons adjustments (such as Bonferroni) at all, and rather treat all the $p$-values as indicating strength of evidence in the Fisherian sence (as opposed to leading to a yes/no decision in the Neyman-Pearson sence).

Statistical tests if you are willing to assume normality

Let's for the moment ignore the issue of normality (see below) and assume that everything is normal. You have the following battery of tests:

  1. For each measure, within-group pairwise comparison between two measurement times is a paired t-test. It will test if the measurements differ between these two times.
  2. For each measure, between-group pairwise comparison for one measurement time is an unpaired t-test. It will test if these two groups differ on this specific measurement.
  3. For each measure, within-group comparison between all three different measurement times is a repeated measures ANOVA. It will test if measurement time has any effect at all.
  4. For each measure, between-group comparison between for one fixed measurement time, is a one-way ANOVA. It will test if groups differ in any way between each other.
  5. For each measure, comparison between all groups and all times is a two-way repeated measures ANOVA. It will test if there is a significant effect of group, significant effect of time, and significant interaction between them.
  6. For all measures, comparison between all groups and all times is a two-way repeated measures MANOVA. It will test if there is a significant effect of group, significant effect of time, or significant interaction between them on all measures taken together.

Note that #1 and #2 can be seen as a post-hocs to #3 and #4 respectively, #3 and #4 can be seen as post-hocs to #5, and #5 can be seen as post-hoc to #6.

[With an additional complication then when these tests are done as post-hocs they use some of the pooled estimates of the "parent" test in order to be more consistent with it; I am not sure though if these procedures exist on the higher levels of the hierarchy.]

So you have a layered structure and you can proceed in the top-down manner from the most general (#6) level down to most specific (#1 and #2) tests and run each next level only if you have significant omnibus effect on the higher level (apologies for the potential confusion; "higher" levels have higher numbers in my list and hence are located on the bottom of it... "top-down" means starting with MANOVA in #6 and going until t-tests in #1 and #2). This should protect you from false positives on the lower level, and so you arguably (!) don't need to do multiple comparison adjustments on the lower level (but as far as I understand, opinions on that differ).

You can also start directly at some middle layer and e.g. run 12 times #5 without doing #6, or 36 times #3 and 36 times #4 without doing #5. In confirmatory framework, you must then apply some multiple comparison correction (such as Bonferroni or rather Holm-Bonferroni). In exploratory framework this is not necessary, see above (example: maybe without adjustment you get $p=0.01$ effect in many different measures and it is very consistent; you are probably looking at a real effect then, but if you make Bonferroni adjustment then everything will stop being significant -- too bad. Instead, in exploratory framework you should rather keep $p=0.01$ as is and use your own expert judgment, but of course at your own risk).

By the way, if your Treatments work at all, you should expect significant effect of interaction in #6 and #5, so these are (hopefully!) almost guaranteed, and the interesting stuff begins at layers #3 and #4. If there is a real danger that both Treatments are as bad as placebo then perhaps you should really start with #6.

Another remark: a more "modern" approach would be to use a linear mixed model (with subjects being a random effect) instead of repeated measures ANOVA, but that's a whole other topic that I am not very familiar with. It would be great if somebody posted an answer here written from a mixed models perspective.

Statistical tests if you are not willing to assume normality

There are ranked analogues of most of these tests, but not of all of them. The analogues are as follows:

  1. Wilcoxon test
  2. Mann-Whitney-Wilcoxon test
  3. Friedman test
  4. Kruskal-Wallis test
  5. ?? (probably does not exist)
  6. ???? (most probably does not exist, but see here)

Additional complication is that post-hocs become tricky. Proper post-hoc to Kruskal-Wallis is not Mann-Whitney-Wilcoxon but the Dunn's test [it takes into account the issue that I mentioned in the square brackets above]. Similarly, proper post-hoc to Friedman is not Wilcoxon; not sure if it exists but if it does it is even more obscure than Dunn's.

Normality testing

It is in general a very bad idea to test for normality in order to decide whether you should use parametric or nonparametric tests. It will affect your $p$-values in an unpredictable way. At least in the confirmatory paradigm, you should decide on the test prior to looking at the data; if you have doubts about normality approximation, then rather don't use it. See here for more discussion: Choosing a statistical test based on the outcome of another (e.g. normality).

In your case, this means that you should use only parametric tests or only nonparametric tests for all measures (unless you have a priori grounds to suspect substantial deviations from normality in only a specific subset of measures; this does not seem to be the case).

In simple cases people often suggest to use ranked tests because they are powerful, simple, and you don't need to worry about the assumptions. But in your case, nonparametric tests will be a mess so you have a good argument in favour of classical ANOVAs. By the way, the histograms that you posted look "normal enough" to me that with your sample size you should not worry too much about them not being normal.

Data presentation

I would strongly advice to rely on visualization as opposed to only listing hundreds of $p$-values in a text or a table. With the data like that, first thing I would do (note: this is very exploratory!), would be to make a giant figure with 12 subplots, where each subplot corresponds to one measure and shows time on the x-axis (three measurements) and groups as lines of different color (with error bars).

Then just stare at this figure for really long and try to see if it makes sense. Hopefully the effects will be consistent across measures, across time points, etc. I would make this figure the main figure of the paper.

If you like, you can then pepper this figure with the results of your statistical tests (mark significant differences with stars).

Brief answers to your specific questions

  1. Yes (almost -- see the caveat about Wilcoxon as post-hoc)
  2. Yes
  3. Yes
  4. Use figures as much as you can.

Word of caution

We would like to know if Treatment 2 (Dietary Supplement 2) has the same effect (or even better) on body composition than Treatment 1, while not having those adverse effects on blood profiles.

To show that Treatment 2 does not have as much adverse effects as Treatment 1, it's not enough to show that there is significant difference between T1 and Controls but no significant difference between T2 and Controls. This is a common mistake. You actually need to show significant difference between T2 and T1.

Further reading:

  • $\begingroup$ One question! I've found Body weight differences to not be significant between groups, but BMI (which is Body weight / Height^2) is significant between group. Would you please elaborate me how to interpret this? $\endgroup$
    – NULL
    Commented Nov 29, 2015 at 18:42
  • $\begingroup$ @Hamed, I am not sure I understand why you are confused about it; is the height significantly different? If so, then no wonder that BMI is. If not, then perhaps body weight and height are both not-significantly but consistently different and when divided by each other yield a significantly different BMI because the effect is combined and "enhanced". $\endgroup$
    – amoeba
    Commented Nov 29, 2015 at 21:55
  • 1
    $\begingroup$ @amoeba This is great. Would you mind elaborating on why one would proceed in a "top-down fashion" with the tests #1 to #6? I think I understand well the point of using Bonferroni corrections when doing multiple tests and the "green beans" comic strip (i.e. if you look hard enough, you will end up finding a significant difference somewhere), but I fail to see how this line of thinking (or another one) explains a "top-down" approach with the tests you laid out. Thanks! $\endgroup$ Commented Dec 1, 2015 at 0:25
  • 2
    $\begingroup$ Thanks @amoeba I think I was probably confused by the numbering of the tests. "Top down" is #6 to #1 in the list, correct? The least specific of the tests seems to be #6, so one would proceed with #6, then #5, then #4, etc, right? $\endgroup$ Commented Dec 1, 2015 at 4:05
  • 2
    $\begingroup$ @Amelio, yes, that's correct. I should edit to clarify. $\endgroup$
    – amoeba
    Commented Dec 1, 2015 at 9:14

It is a multi layered methodological onion to be peeled. I will only be able to deal with the top layers both because of the lack of time and the lack on knowledge. I will base this answer in the very clear statement of the goals of the analysis: in bold in the OP:

We would like to know if Treatment 2 (Dietary Supplement 2) has the same effect (or even better) on body composition than Treatment 1, while not having those adverse effects on blood profiles.

1) There is no need for the control group - you want to compare two groups Treatment1 and Treatment2 - this is good because you can do 2 group tests without multiple comparisons - (at least in principle) instead of multiple group tests _ post hoc tests

2) Let us assume that you have a single measure of body composition, say B. You want to show that T2 (treatment 2) is at least as good as T1 on the B measure.

A big problem here. All the tests you mentioned are tests to show that one group of measures is different than another, not to show that it is at least as good. Yes, you can use a standard 2 group test (say t-test - forget about Gaussian and non-Gaussian data for a while) and show that the B measures for T2 are significantly different (and better) than that of T1. If you are lucky, and you get the significant difference than you can show that T2 is better than T1 and thus, at least as good. But if you are not lucky, then what did you get - the fact that the p-value is high does not tell you that the two sets of measures are the same (and thus T2 is as least as good as T1) , it tells you dont have enough data to show that there is a difference!!

So what you need for the B measure is a non-inferiority test (or an equivalence test). I will not get into it - there are many answers in CV on equivalence tests. But my point 1 above is important because the non-inferiority tests I know (TOST for example) only work with two groups!

3) Let us assume you have only one blood measure (C). You want to show that T2 is better than T1 on the C measure, and here standard tests - the ones that show difference - are the appropriate tool. You show that the difference on the measure C for T1 and T2 are significantly different (and that T2 is better) and thus that T2 has significantly better on the blood profile.

4) Another problem is that you dont have a single measurement of B (and C) for each subject. You have 3 measurements at 3 different times

I dont really know what to do with the 3 measurements per subject. Notice that this is not a within-subject that matters to the research question - the 3 measurements are on the same subject but we are comparing the set of subjects in T1 and T2, and the subjects in T1 and T2 are not paired or the same.

I guess that I would treat the 3 timed measurements as 3 independent measurements to get a better estimate of the true value of B (and C) for each subject. Thus I would just average the three timed measurements into a single one. (I understand one would lose the information on variability by averaging the data but it is unclear to me where this information of variability of the B measurements would be useful for the research question).

4) The next problem is that there in no single B measure for body composition there are many different measures such as Body Weight, Body Mass Index, Body Fat Mass,which are probably correlated). Let us call them Ba, Bb, Bc and so on. (Notice that this is not the 3 measurements in time for each subject discussed above, they are different measures - I used measurements in the item above, and measures here)

You can run the procedure described so far (up to item 3 above) for each body measure (average the 3 measurements per subject, perform a non-inferiotity test on the two sets of data) on each Ba Bb Bc measure, and report the results. The same for all the blood measures Ca and Cb, and so on. But then you are making a lot of comparisons and test. In this example there would be 5 tests results (Ba Bb Bc Ca and Cb). Therefore you should also have a multiple comparison procedure to adjust the p-values!! (This is very uncommon - people usually do not do p-value adjusting for different tests - only for a single, multiple group test - but they should do it).

On the other hand, the measures Ba Bb and Bc are very correlated, and thus the results of the tests are not independent - and I dont know how to do the p-value adjustement (Notice that the Bonferroni correction assumes that each of the tests are independent - exactly the opposite of the situation here).

I will stop the answer at this time. Hopefully more knowledgeable CV contributors will be able to provide better answers, specially to points 3 and 4 above which are at the limit of my knowledge.

  • $\begingroup$ This is a good answer $\endgroup$
    – Glen_b
    Commented Dec 1, 2015 at 3:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.