I have two questions on interpreting the statistical analysis of data collected from production runs to produce an experimental material. We made four production runs, with some changes in each run. Each run produced three pieces of material, and each piece of material was tested in multiple places to measure the property of interest. So, the measurement results look like this:
Run 1 Piece A -- 25 measurements;
Run 1 Piece B -- 25 measurements;
Run 1 Piece C -- 25 measurements;
Run 2 Pieces A, B, and C -- 25 measurements on each
Run 3 Pieces A, B, and C -- 25 measurements on each
In order to determine if there existed a significant variation among the production runs, I performed Duncan's test using three groups of three replicates, taking the average measurements from one piece as one “replicate” (i.e. the value for group "Run 1" and replicate "Piece A" was the average of the 25 measurements). So the first question is: is it valid to take the average of the measurements from one piece and call it one replicate? I could say I have 9 groups with 25 replicates each, but my goal is to compare batches and not pieces independently.
The Duncan’s test showed Run 2 to be significantly different from the other two runs. To complicate things, we made a fourth run:
Run 4 Piece A -- 9 measurements;
Run 4 Piece B -- 9 measurements;
Run 4 Piece C -- 9 measurements.
The fourth group had much higher averages and standard deviations from the others. When I repeat Duncan’s test adding in this fourth group of data, the test shows the fourth group to be significantly different from the other three, but the other three are no longer different from each other.
So, I am faced with interpreting the significant difference. With only three runs Run 2 was significantly different from Runs 1 and 3. After doing the fourth run, Run 2 was no longer different from Runs 1 and 3, just different from Run 4. I understand mathematically what occurred – that the results from Run 4 were so far different that they pushed the first three runs together. But what is the practical interpretation (second question)? Is Run 2 really different from Runs 1 and 3?
Any thoughts on this problem are greatly appreciated!
