Stratify the analysis if Levene's test fails

Question

I have a question about the correctness of a statistical analysis.

I have a variable called L, which is the log of the number of bacteria present in some foods. L is a function of the treatment T (four levels), the food F (7 levels) and the bacteria B(5 levels). Each observation comes in three replicates, meaning that for each value of (T, B, F), three independent measurements have been performed. I need to know which groups share the same variance in L.

Each group is labelled by a particular treatment, bacteria and food, i.e. each group is identified by the triplet (T, B, F). If I compare the variance among each group (T, B, F) I have only three points to estimate the variance from (the three replicates) and the estimate is not really reliable.

I have no problem in assuming that the variance among treatments is the same, so that I can pool them together and now my groups will be identified by (B, F) and each group contains 3 replicates * 4 treatments = 12 obsevrations. Now I can estimate the variance in each group and use a Levene's test to test for homogeneity. The first test asks if all the groups (B, F) have same variance. I find that the p value is $10^{-10}$ so I can say that the variances are different.

The next steps I am not sure can be done. I want to understand which groups have the same variance. I stratify the analysis by food and for each F I do a Levene test where I test the homogeneity among the groups labelled by (B). I find most of the bacteria B have very high p values (except for two). So for these bacteria I can consider that the variances are the same across foods.

Does this make sense?

Answer 1

I need to know which groups have the same variance in L.

Emphasis mine. This is tricky, most statistical tests are concerned about telling you when things are different, not when they're the same. In fact, even when a test tells you that there's "no significant difference" that doesn't mean that they're "the same." It might mean they're "the same" or it might mean you don't have enough data to prove they're different.

Each group is identified by the triplet (T, B, F)

I'm going to set up some notation to use for the rest of the answer. Let's let each group as defined above be denoted $G_{tbf}$ with $t$ running from $1$ to $4$; $b$ running from $1$ to $5$; and $f$ running from $1$ to $7$.

I have no problem in assuming that the variance among treatments is the same

I do. There ought to be a test for this which shows $Var(G_{1..}) = Var(G_{2..}) = … = Var(G_{4..})$. To continue with the answer, I'm going to assume you did the test, which is why you're comfortable making the assumption.

I can pool them together and now my groups will be identified by (B, F) and each group contains 3 replicates * 4 treatments = 12 obsevrations.

This is another tricky thing to do, since there might be interaction between $t$ and $b,f$. The fact that you got a P-value so low might be due to differences in variation caused by interaction with $t$, since they've been pooled, there's no way to know.

I stratify the analysis by food and for each F

I would avoid Post-Stratification at this point in the analysis. There's a penalty for it.

What to do?

The good news is that the fact the number of replicates (call it $R$) is 3 doesn't hurt you as much as you think it does. You can very easily perform an ANOVA to get variance estimates for $t$, $b$, and $f$. You can even get variance estimates for the variance between all three factors. With those pieces of information, you ought to be able to tell which variances are not the same at the very least.