How to analyze variance from repeating a test with the same unit vs variance across units?

Question

Suppose I want to buy a CNC machine to automate guitar-making, for instance. I have access to a warehouse of CNC machines that are all reported to be the same model and advertised as having equal performance (e.g. 100 guitars/day) and I can test them in any way I want so that I can ultimately choose one of them. Unfortunately, I suspect that they don't all perform the same as advertised and I want to prove that their differences aren't just due to normal variation.

How do I show that the variance across machines is significant when compared to the variance of running a single machine multiple times?

Another way to phrase this is, how do I show that a single machine is consistent with itself (when comparing repeated tests) vs. showing that when testing various machines, the results are more varied?

I suppose a simple variance across machines would tell me how much they all differ from each other, but I want to rule out that this variance is the same when running the same machine multiple times.

Based on this question, I suspect a Levene's test is what I want, but would ANOVA also show what I want if I expect that the overall mean of the machine performances is equal to the advertised performance?

Answer 1

I think that the solution you need is basically a Gage R&R (which is itself an ANOVA). Say you have 10 CNC machines (# of parts), each will make 10 parts (# of operators), and you will measure each part 3 times (# of replicates). The analysis of this will break down the total variance (accross all the parts from all machines and all replicates) as the sum of the variance due to the parts (the machines in your case), the variance due to the in-machine variability (the variance of individual machine in your case), and the variance in the replicates. You can use only 1 replicate if you believe that your measurement method is much more reproducible/reliable than the other 2 variances. And yes, this is basically a (2-way) ANOVA. And no, you can not use Levene: all it can tell you is that variances are equal (or not): you can use it to compare the variances of the machines, but not to see if the within machine variance is greater than the between machine variance (seems to be what you want). That is where you need Gage R&R. Also Levene will tell you (like all omnibus tests) that at least 2 variances are significantly different, but you do not know which 2, nor if there are more than 2 which are different. To do that, you will need to do multiple comparisons, and use an appropriate multiple comparison correction (Sidak?) to avoid alpha error inflation... For more details on GR&R, maybe look at https://www.spcforexcel.com/knowledge/measurement-systems-analysis/three-methods-analyze-gage-rr-studies/ Most statistical software (e.g. Minitab) will have tools to perform GR&R analysis: feed it the data, you will get your results.