# Splitting groups for statistical comparisons

I have a question that I cannot answer based in my biological background.

I conducted one experiment with 5 replicates and 4 treatments.

There were two goals: the first goal was to elucidate if our new method during fermentation could give us better results in relation to a standard. The second goal was to investigate if flask size affects the fermentation.

Well, when I performed one-way ANOVA with all treatments, I got the following with Tukey's test (letters mean the statistical grouping - 95% confidence):

• Standard in small flasks (A)
• Old method in small flasks (AB)
• New method in small flasks (B)
• Old method in big flasks (B)

When I removed the treatment with the big flasks and ran one-way ANOVA, I got the following results in Tukey's test (letters mean the statistical grouping - 95% confidence):

• Standard in small flasks (A)
• Old Method in small flasks (B)
• New Method in small flasks (B)

The post hoc test without one treatment (Old Method in Big Flask) makes sense, but my question is if it is statistically fair to analyze as two different experiments:

• Old method in small flask VERSUS Old method in big flask
• Standard in small flask VERSUS Old method in small flask VERSUS New method in small flask

In short, I'm asking if I could run two one-way anova with post hoc tests as if I were analyzing two different experiments (subsets of the single experiment). I know that it may sound dumb, but I can't figure out it alone.

• What is your study design? It seems (?) like you have a single study with 2 factors: Method and Flask Size. Method has 3 possible levels: Standard, Old and New. Flask Size has 2 possible levels: Big and Small. It's not clear if you considered all possible combinations of these 2 factors (with 5 replicates for each) or just a specific subset of these combinations. If the latter, can you clarify in your post what specific combinations you used in your study? May 1, 2019 at 1:58
• It is not clear to me - and likely to others on this forum - what your conditions are. Also, not clear what kind of Anova you used - one way, two way with main effects only, two-way with main effects and interaction. More information is needed before being able to make sense of results. May 1, 2019 at 2:07
• @IsabellaGhement, thank you for helping me. I edited my post. Is it better now? May 1, 2019 at 2:20
• I'm evaluating ethanol titers for each replicate in minitab. May 1, 2019 at 2:30
• I did not run two-way Anova because there is only one treatment with big flask. So, I thought about making subsets and I was wondering if this would be fair. Since I conducted the experiment with all treatments at the same moment, must I compare all treatments together in a single One-way Anova and Tukey's test? May 1, 2019 at 2:49

Thank you for your edits. Just as I suspected, you have a single experiment with two factors: Method and Flask Size. Your experiment includes 5 replicates for each of these combinations of factors:

Method      Flask Size
------      ----------
Standard    Small
Old         Small
New         Small
Old         Big


I find it easier to think of your problem in a linear regression setting, so my answer will reflect that.

Because you haven't included all possible combinations in your study, you can only consider a linear regression model with main effects for Method and Flask Size. (Alternatively, you could consider a two-way ANOVA model with main effects for these factors.)

Your linear regression model would quantify the main effect of Method via two dummy variables and the effect of Flask Size via a single dummy variable. The model would look like this:

Outcome = beta0 + beta1*DummyOld + beta2*DummyNew + beta3*DummyBig + error


where

DummyOld = 1 if a replicate comes from Old method and 0 if that replicate comes from either of the remaining two methods

DummyNew = 1 if a replicate comes from New method and 0 if that replicate comes from either of the remaining two methods


and

DummyBig = 1 if a replicate comes from a big flask and 0 if that replicate comes from a small flask


Note that this model assumes that the effect of Method on the outcome is the same for each Flask Size. It also assumes that the effect of Flask Size on the outcome is the same for each Method. (You can't assume different effects because your study does not include all possible combinations of Method and Flask Size.)

Your study was set up to answer the following research questions (translated in statistical language):

1. For a particular flask size, is the mean value of the outcome smaller/larger for the new method compared to the standard method? (Pick smaller or larger, depending on what it means for the new method to be "better" than the standard method.)

2. For a particular method, is there a difference in the mean value of the outcome between big flasks and small flasks?

These are the only hypotheses you need to carry out and because they are planned, you can test them directly, without conducting an omnibus F-test for testing the overall significance of the model.

You can address the first research question by testing the null and alternative planned hypotheses:

Ho: beta2 = 0 versus Ha: beta2 < 0 (or beta2 > 0, depending on how you define better)

You can address the second research question by testing the null and alternative planned hypotheses:

Ho: beta3 = 0 versus Ha: beta3 != 0.

So all you need to do is perform these two tests of hypotheses based on the suggested linear regression model. Some people don't adjust the p-values for such tests for multiplicity, arguing that the hypotheses being tested were planned at the study design stage. Personally, I prefer to adjust them for multiplicity and report both adjusted and unadjusted p-values. (Multiplicity arises from performing more than one test from the same data - in your case, two tests.)

You can also report a one-sided confidence interval for beta2 and a two-sided confidence interval for beta3.

The beauty with the approach suggested here is that:

1. It acknowledges you have a single study;

2. It uses a model which can be used to test two sets of pre-planned hypotheses, inspired by your two research questions;

3. It obviates the need for performing an omnibus test of significance of the full model.

I do not use Minitab but I provided enough detail here for you to be able to perform the suggested analyses in Minitab.

• Thank you very much for your answer! I have never analyzed data this way, so let me ask you some questions and then I am going to run it as you suggested. I should use literally 1 or 0 for the dummy variables? What will be my beta values? For example, in my One-way Anova I used the ethanol titer measured in determined time of fermentation. Should it be y=ethanol titer and beta=time (it is the same for every flasks)? Could I conduct the same approach with two-way Anova even without a complete delineation (combinations of the two factors)?Sorry to bother again. May 1, 2019 at 12:05
• You didn't mention anything about time in your original post, so you need to provide more details about how you actually conducted the experiment. May 1, 2019 at 19:57
• Sorry, I didn't see it as another variable because it is fixed within the ethanol measurement. I did not consider it even in one-way ANOVA and Tukey's test. I just asked because I did not get what I should use for beta values. May 1, 2019 at 21:51
• I don't know enough about your study to feel confident that my advice actually makes sense. If it was reasonable to proceed as I described in my post, then the betas would be unknown coefficients estimated from the data. In your linear regression model, you would just enter the outcome variable (ethanol titer) and the dummy variables suggested in my post. The betas would be estimated by the software. Since I don't really understand how time plays a role in your study, I would say: take my response with a grain of salt, as it doesn't capture any information on time. May 1, 2019 at 23:09