NOTE: Revised question following suggestions from others. Thank you for taking the time to help me refine this question to make it answerable!


I want to compare the performance of two versions of a device: V1 and V2. I have done two pilot studies. I used a linear mixed-effect model and found 100% of my variance is attributable to measurement error. This matches my expectations, but how do I deal with it?

The performance of the device is measured as the input force required to do some work on a test subject in a standardized test. Each device is attached to a standard test rig which applies the input force in the same way every time. The input force is applied to the device and the device translates this into an output force which stuffs the test subject into a hole. The test subject is essentially a bag filled with viscous fluid.

There is an element of randomness in the way the fluid moves and the bag wrinkles as it is pushed. These random events do affect the input force needed to push the bag through the hole. This can be expressed as $x^{*} = x+\Delta$. Where, $x$ represents the theoretical input force required if there were no fluid movement or bag wrinkling, and $\Delta$ represents the (random) additional force required to overcome these dynamic effects. I can only measure $x^{*}$ - there is no way to measure $x$. Measurement of $x^{*}$ is continuous.

To compare performance, I want to test:
$H_{0}: x_{V2}^{*} \ngtr x_{V1}^{*}$
$H_{A}: x_{V2}^{*} > x_{V1}^{*}$

The minimum difference between groups that I want to be able to detect is 1.5 units of input force.

Note that I have plenty of V2 devices available to test and do pilot studies on. I have 10 V1 devices, but they are out of production so it's difficult and expensive to obtain more (long lead time and high cost associated with setting up and running more). I may be able to scrape together another 5 or so V1 devices from other sources, but I don't know for sure. Testing is non-destructive and it's fairly easy to get multiple repeat measures off a single device.

What I've done

I did a pilot study on 10 V2 devices and I measured the input force on 2 separate but identical test subjects with 2 repeat measures per combination (total of 40 runs). Below are the results from trying to fit this to a linear mixed effects model in Minitab: ≈90% variance is attributed to error, ≈10% is attributed to the test subject, and 0% is attributed to the device. This generally matches my expectations because 1) the principles of operation of the device and the test rig do not suggest there would be much (if any) detectable difference between devices; 2) I expect randomness (error) due to random factor $\Delta$ mentioned above. I actually suspect the ≈10% attributed to test sample is an artifact of not having enough repeat measures (labeled replicate in the table). I am doing a second pilot study to investigate this further.

Factor Type Levels Values
Device Random 10 1,2,3,4,5,6,7,8,9,10
Test Subject Random 2 A,C
Source Variance % of Total SE Var Z-Value P-Value
Device 0.00000 0.00% * * *
Test Subject 0.160500 10.65% 0.322545 0.497604 0.309
Device*Test Subject 0.000000 0.00% * * *
Error 1.346250 89.35% 0.308851 4.358899 0.000
Total 1.506750
-2 Log Likelihood = 127.180857
S R-sq R-sq(Adj.) AICc BIC
1.16028 7.46% 7.46% 136.36 141.84

Here is the data from the Pilot Study:

RunOrder Device Test Subject Replicate Helper Helper2 Result
1 5 A 1 A.1 5.A.1 8.5
2 9 C 1 C.1 9.C.1 9.2
3 6 C 1 C.1 6.C.1 9.8
4 7 A 1 A.1 7.A.1 7.6
5 7 C 1 C.1 7.C.1 6.6
6 2 A 1 A.1 2.A.1 8.2
7 1 C 1 C.1 1.C.1 8.4
8 5 C 1 C.1 5.C.1 7.3
9 4 A 1 A.1 4.A.1 8
10 3 A 1 A.1 3.A.1 7.1
11 4 C 1 C.1 4.C.1 9.4
12 5 C 2 C.2 5.C.2 9.7
13 6 A 1 A.1 6.A.1 5.9
14 8 A 1 A.1 8.A.1 9.5
15 2 C 1 C.1 2.C.1 7.1
16 10 A 1 A.1 10.A.1 7.8
17 3 C 1 C.1 3.C.1 7.5
18 2 C 2 C.2 2.C.2 7.6
19 5 A 2 A.2 5.A.2 6.8
20 8 C 1 C.1 8.C.1 8.6
21 8 C 2 C.2 8.C.2 8.3
22 3 C 2 C.2 3.C.2 10.3
23 7 A 2 A.2 7.A.2 5.3
24 9 A 1 A.1 9.A.1 8.5
25 9 C 2 C.2 9.C.2 8.6
26 1 C 2 C.2 1.C.2 8.6
27 10 A 2 A.2 10.A.2 6.9
28 1 A 1 A.1 1.A.1 6.3
29 10 C 1 C.1 10.C.1 6.8
30 4 C 2 C.2 4.C.2 6.4
31 6 A 2 A.2 6.A.2 7.5
32 8 A 2 A.2 8.A.2 5.6
33 10 C 2 C.2 10.C.2 6.1
34 9 A 2 A.2 9.A.2 7.4
35 7 C 2 C.2 7.C.2 7.4
36 6 C 2 C.2 6.C.2 7.8
37 4 A 2 A.2 4.A.2 7.7
38 2 A 2 A.2 2.A.2 8.7
39 1 A 2 A.2 1.A.2 8.3
40 3 A 2 A.2 3.A.2 6.4

Questions I have

Question 1 - How to handle the measurement error? Are repeat measures the best way?

How do I deal with this measurement error? It seems to me the best way to deal with this inherent randomness would be to take repeat measures for each device. Is that an appropriate and valid approach? Is it the right approach?

Question 2 - How many repeat measures are needed?

Assuming repeat measures are an appropriate way to deal with this variability, how do I decide how many repeat measures are needed? i.e., is 2 enough? 3? 5? 10? How can determine this?

Second Pilot Study

One of my test subjects was damaged, so I had to switch to another pair of test subjects. These have a larger volume, which makes the absolute values in this test higher than the previous test. I don't really care about absolute values, I just want to be able to compare my two groups.

Results from LMM:

Factor Type Levels Values
Device Random 10 1,2
Test Subject Random 2 A,C
Source Variance % of Total SE Var Z-Value P-Value
Device 0.00000 0.00% * * *
Test Subject 0.00000 0.00% * * *
Device*Test Subject 0.00000 0.00% * * *
Error 1.345297 100.00% 0.247689 5.431390 0.000
Total 1.345297
-2 Log Likelihood = 189.029348
S R-sq R-sq(Adj.) AICc BIC
1.15987 0.00% 0.00% 197.77 205.34


StdOrder RunOrder Device Test Subject Replicate Helper Helper2 Result
8 1 2 B 1 B.1 2.B.1 10.7
6 2 1 B 1 B.1 1.B.1 11.7
58 3 1 B 2 B.2 1.B.2 10.2
60 4 2 B 2 B.2 2.B.2 11.5
47 5 2 A 1 A.1 2.A.1 11.6
45 6 1 A 1 A.1 1.A.1 10.7
13 7 1 A 2 A.2 1.A.2 9.7
15 8 2 A 2 A.2 2.A.2 10.6
1 9 1 A 3 A.3 1.A.3 10.7
3 10 2 A 3 A.3 2.A.3 9.9
23 11 2 A 4 A.4 2.A.4 10
21 12 1 A 4 A.4 1.A.4 11.9
25 13 1 A 5 A.5 1.A.5 10
27 14 2 A 5 A.5 2.A.5 11.3
55 15 2 A 6 A.6 2.A.6 10.4
53 16 1 A 6 A.6 1.A.6 10.5
20 17 2 B 3 B.3 2.B.3 10.1
18 18 1 B 3 B.3 1.B.3 10.9
24 19 2 B 4 B.4 2.B.4 8.9
22 20 1 B 4 B.4 1.B.4 11.4
33 21 1 A 7 A.7 1.A.7 9
35 22 2 A 7 A.7 2.A.7 11.2
49 23 1 A 8 A.8 1.A.8 10.9
51 24 2 A 8 A.8 2.A.8 8.3
10 25 1 B 5 B.5 1.B.5 10.3
12 26 2 B 5 B.5 2.B.5 11.5
43 27 2 A 9 A.9 2.A.9 9.2
41 28 1 A 9 A.9 1.A.9 9.6
52 29 2 B 6 B.6 2.B.6 9.5
50 30 1 B 6 B.6 1.B.6 9.8
40 31 2 B 7 B.7 2.B.7 12.1
38 32 1 B 7 B.7 1.B.7 9.7
56 33 2 B 8 B.8 2.B.8 10.7
54 34 1 B 8 B.8 1.B.8 10.2
29 35 1 A 10 A.10 1.A.10 11.8
31 36 2 A 10 A.10 2.A.10 11.2
44 37 2 B 9 B.9 2.B.9 8.2
42 38 1 B 9 B.9 1.B.9 10.6
32 39 2 B 10 B.10 2.B.10 10.2
30 40 1 B 10 B.10 1.B.10 8.6
28 41 2 B 11 B.11 2.B.11 9.6
26 42 1 B 11 B.11 1.B.11 9.1
7 43 2 A 11 A.11 2.A.11 11.3
5 44 1 A 11 A.11 1.A.11 8.7
57 45 1 A 12 A.12 1.A.12 11.9
59 46 2 A 12 A.12 2.A.12 11.2
39 47 2 A 13 A.13 2.A.13 11
37 48 1 A 13 A.13 1.A.13 10.7
17 49 1 A 14 A.14 1.A.14 10.4
19 50 2 A 14 A.14 2.A.14 8
36 51 2 B 12 B.12 2.B.12 10.4
34 52 1 B 12 B.12 1.B.12 9.8
16 53 2 B 13 B.13 2.B.13 8.3
14 54 1 B 13 B.13 1.B.13 13.3
48 55 2 B 14 B.14 2.B.14 11.1
46 56 1 B 14 B.14 1.B.14 9.8
4 57 2 B 15 B.15 2.B.15 9.4
2 58 1 B 15 B.15 1.B.15 8.3
11 59 2 A 15 A.15 2.A.15 9.2
9 60 1 A 15 A.15 1.A.15 12.7
  • $\begingroup$ Isn't the test a source of failure for these sampling units, contributing to non-independence of events beyond repeated measure? $\endgroup$
    – AdamO
    Feb 8 at 19:49
  • 1
    $\begingroup$ "I did a pilot study using 10 specimens of widget.b (I expect the variance to be the same between widget.a and widget.b) with 2 repeat tests on each of 3 test subjects." What sort of tests? What is a subject and what is a specimen? What is a widget? $\endgroup$ Feb 14 at 20:57
  • 1
    $\begingroup$ A related question might be stats.stackexchange.com/questions/606535/… $\endgroup$ Feb 14 at 21:27
  • 3
    $\begingroup$ @CBRF23 it is helpful to be more specific. A test is so general. What sort of variables are the output? And 'widgets', 'subjects', 'specimens', that makes it confusing what you mean. What are the relationships between these terms. Which terms relate to random effects, which to fixed effects, what is the response variable, which variable is used in creating pairings, all such aspects are unclear when you are not specific about what your test is. Currently I am puzzled about a widget and what it means to have specimens from a group of widgets. I onlyknow widget as a small piece of software . $\endgroup$ Feb 15 at 17:53
  • 2
    $\begingroup$ I agree with @SextusEmpiricus about the clarification of the definition of variables. Furthermore, "repeated measures" has a special meaning in statistical modeling, which refers to multiple measurements of the same participant that constitute longitudinal data and motivate models of mixed (i.e., random and fixed) effects. Your use of "repeated measures" does not align with this paradigm. $\endgroup$
    – DrJerryTAO
    Feb 16 at 12:29

2 Answers 2


Repeated testing of one widget will allow you to determine the magnitude of measurement variation (noise) and, maybe, if there are time or test-dependent changes in the performance. Those things are important, but they should typically be determined in advance of performing an experiment to determine the effect of the coating. The statistical test of the effect of coating should be based mostly on the variation between widgets rather than measurements, as the test intervention (coating) is per widget, not per measurement.

In many settings the measurement noise is small compared to the inter-widget variability, and that means that there is no substantial benefit to measuring more than once. However, if your system yields measurement noise that is too large to ignore then averaging multiple measurements is a good way to deal with it. The number of measurements to average depends on the measurement variability relative to the inter-widget variation.

In your statistical test of the coating, an independent (i.e. unpaired) t-test sounds more appropriate unless you have more relatedness within your proposed pairs than is apparent to me.

For power calculations for the effect of coating you need to first have a design in mind so that you can input the expected standard deviation into the calculation. That means that you need preliminary tests of the measurement noise and a decision on how many measurements to average. You also need either a realistic estimate of the effect of the coating to serve as the 'alternative' hypothesis size for the calculation, or an effect size that you would not want to erroneously fail to detect.

If measurements are cheap (inexpensive, easy, quick) then make lots and average them within a widget. If they are expensive then it is worth your while to perform serious preliminary characterisation of the inter-measurement variation and inter-widget variation.

  • $\begingroup$ Measurements are cheap, but widgets are not. Historic testing was done on coating A. Looking to see if results are meaningfully different for coating B. Improved performance in this test is allowed. Worse performance in this test would require further testing of coating B at great expense. Historically, there is high variation between measurements - hence the desire to do multiple repeat measurements. Not sure how to best determine how many repeat measurements are needed, or how to square that with minimum sample size. Sorry if original question was unclear. $\endgroup$
    – CBRF23
    Feb 8 at 21:10
  • 1
    $\begingroup$ There are dangers involved with using historical data to compare with new measurements, particularly if the measurements are less than consistent. For thinking about how many measurements to average, be aware that the benefit of each successive measurement declines in some proportion to the square root of the number of measurements. $\endgroup$ Feb 9 at 4:19
  • $\begingroup$ Thanks Michael Lew! Historic data wouldn't be used for comparison, just to help with planning (i.e., historic variance used to calculate power of detection and sample size). New widgets will be tested directly against each other for the T-Test. Your answer helps explain some of the considerations in planning, but I'm still not sure how to determine the number of individual samples and replicates that would be needed. I'm open to other tests if a Paired-T is not appropriate (e.g., MANOVA, regression, etc.) Low part-to-part variance/high measure-to-measure variance has me thrown. $\endgroup$
    – CBRF23
    Feb 9 at 19:38
  • $\begingroup$ @Michael_Lew - I've revised the question and started a bounty. You've answered question 1 for me (average the repeats), but could you expand on why it's okay to compare averages of averages in a comparison test? Re: Question 2 - you say "The number of measurements to average depends on the measurement variability relative to the inter-widget variation." How do I take my knowledge of inter-widget variation and use that to determine an appropriate number of repeat tests? $\endgroup$
    – CBRF23
    Feb 13 at 20:53
  • 1
    $\begingroup$ I can't provide a revised answer that will help because I do not know how best to set up the test. I could do a series of simulations with pseudo data to find the performance of various approaches, but it should probably be you doing that rather than me. $\endgroup$ Feb 17 at 20:45

I want to compare the performance of two versions of a widget using a 2-sample comparison test. H_0 = both versions perform the same, H_1 = widget.b performs worse than widget.a.

Your hypotheses might be ill defined. They currently represent a boundary case where the null and alternative hypotheses do not complement to an entire continuous region of parameter space. A more common hypothesis definition is $H_0$: Widget B performs no worse than Widget A ($y_\text A \leqslant y_\text B$) vs. $H_1$: Widget B performs worse than Widget A ($y_\text A > y_\text B$). In this setting, we aim to find evidence that A outperforms B. It is very important to define what performance means and how it is measured. Is $y$ continuous, nonnegative, strictly positive, integer, or a duration? Response variables of different types require different modeling methods even if the hypotheses are about means.

I have good reason to suspect there is inherent variability in the performance metric that is being measured. I want to include multiple replicates for each specimen to try to address this.

As Sextus Empiricus pointed out, you better clarify what 'widgets', 'subjects', 'specimens', 'replicates', 'repeat measures', and 'inherent variability' mean. Having variability in the response variable is natural and necessary to conduct statistical tests and analysis. Without variability, no statistical analysis is necessary.

Question 1 - how many replicates. I'm not sure how to decide how many repeat measures (replicates) I should make with each specimen. Is 2 enough? Is 3? 10? What process (with reputable source) can I use to determine how many repeat measures I should include?

It appears that you are talking about sample-size determination. Power analysis is used to find the required sample size at given values of significance level, power, and effect size in a specific test method. The sample size to obtain ideally should be neither more nor less than this calculated requirement, as it would deflate and inflate p values, respectively.

Question 2 - how to treat the replicates in the comparison test How does one go about incorporating replicates into the sample data for comparison testing?

Measuring the outcome multiple times on each participant can provide data to estimate within-participant variation, which makes the effect estimation more precise, standard errors smaller, and statistical tests more powerful. With longitudinal data, we need to use mixed-effect modeling. Many different experimental design can emerge: (1) one participant measured only once, (2) one participant measured multiple times but on only one version of one of the two widget, (3) one participant exposed to multiple versions of the same widget, (4) one participant exposed to both widgets, ect. With mixed-effect modeling, no need to average the responses before estimation.

I did a pilot study using 10 specimens of widget.b (I expect the variance to be the same between widget.a and widget.b) with 2 repeat tests on each of 3 test subjects. As expected, most of the variance (92%) comes from the repeat measurement (i.e., it's inherent to the measurement), with only 8% of the variance attributable to the differences in specimens. This makes sense to me as the characteristics that define the performance of the widget in this test do not vary much from widget to widget.

In your pilot study, you had 60 rows from three participants, each of whom had 20 measurements, two per widget version. Since the ANOVA table is not built from a mixed-effect model, it may decompose the variation incorrectly and deliver a wrong impression.

Repeat measures are cheap, but tested components cannot be placed back into inventory (i.e., it's a non-destructive test, but inventory value is reduced by testing). I have many version b widgets available and I can sample as many as I need (within reason - more than around 60 would be difficult just due to time constraints). Version a widgets are expensive and have a long lead time. I have up to 10 available to test, but would prefer to consume less if possible.

It is okay to include many versions of a widget; in fact, when using fixed-effect estimators, both the number of participants and the number of repeated measurements on each participant, which are likely done across multiple widget versions, need to be large to get consistent estimates. In a real experiment, you will likely model both widget versions and participant identities as random effects.

My power and sample size calculations, based on the sample mean and sample s.dev from my pilot study, indicate I >90% power with n=4 and >97% power with n=5.

Your power analysis look suspicious. A sample size that small does not look right. Comparing means does not mean that t, z, or ANOVA are the best choice. You need to choose the correct model before power analysis, and that depends on the response variable type and experimental design.

  • 2
    $\begingroup$ "Without variability, no statistical analysis is possible." I think it's clearer to say, "Without variability, no statistical analysis is necessary." :) $\endgroup$
    – num_39
    Feb 17 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.