I have two questions on interpreting the statistical analysis of data collected from production runs to produce an experimental material. We made four production runs, with some changes in each run. Each run produced three pieces of material, and each piece of material was tested in multiple places to measure the property of interest. So, the measurement results look like this:

Run 1 Piece A -- 25 measurements;
Run 1 Piece B -- 25 measurements;
Run 1 Piece C -- 25 measurements;

Run 2 Pieces A, B, and C -- 25 measurements on each

Run 3 Pieces A, B, and C -- 25 measurements on each

In order to determine if there existed a significant variation among the production runs, I performed Duncan's test using three groups of three replicates, taking the average measurements from one piece as one “replicate” (i.e. the value for group "Run 1" and replicate "Piece A" was the average of the 25 measurements). So the first question is: is it valid to take the average of the measurements from one piece and call it one replicate? I could say I have 9 groups with 25 replicates each, but my goal is to compare batches and not pieces independently.

The Duncan’s test showed Run 2 to be significantly different from the other two runs. To complicate things, we made a fourth run:

Run 4 Piece A -- 9 measurements;
Run 4 Piece B -- 9 measurements;
Run 4 Piece C -- 9 measurements.

The fourth group had much higher averages and standard deviations from the others. When I repeat Duncan’s test adding in this fourth group of data, the test shows the fourth group to be significantly different from the other three, but the other three are no longer different from each other.

So, I am faced with interpreting the significant difference. With only three runs Run 2 was significantly different from Runs 1 and 3. After doing the fourth run, Run 2 was no longer different from Runs 1 and 3, just different from Run 4. I understand mathematically what occurred – that the results from Run 4 were so far different that they pushed the first three runs together. But what is the practical interpretation (second question)? Is Run 2 really different from Runs 1 and 3?

Any thoughts on this problem are greatly appreciated!

  • $\begingroup$ Thanks for the edit, Christophe. It makes for clearer presentation. $\endgroup$ – user12125 Aug 20 '12 at 18:18

This points out, again, one of the many problems with significance testing.

I would say that the way to interpret the results is to concentrate on measures of effect size. These do not change when you add a fourth group as described.

  • $\begingroup$ I have not dealt with effect size before, but it appears to be a good idea. I will examine that. Thanks! $\endgroup$ – user12125 Aug 17 '12 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.