# Number of observations to study the reproducibility

I'm running an experiment which is about to investigate the influence of gases on the resistance of sensors. Since this is chemistry and gases are experimentally rather hard to handle, I would like to study how well the experimental setup itself is performing. To start simple, I would like to get started with any potential deviations between the runs, so, precisely this means I would do the following:

The system and its components do not change and I flood the volume with gas. Then I measure. Afterwards I air the volume and wait several hours longer than necessary. Then I flood the volume with gas and measuer again. I air again. Over and over.

Here is my first question: How often should I do this? 5 times? 10? 30?

Despite from that, is there an another / more "statistical" way?

• It will be hard to answer this question unless there you provide more information about the variability among resistance measurements under the same conditions, whether you expect the various gases to affect that variability, and how big the differences among gases that you wish to detect are. It would also help to have information about how many gases you wish to test in this way.
– EdM
Jan 29, 2020 at 15:04
• Using the same operator, on the same set-up, under the same conditions with the same sample is a Repeatability test. I can't find any good references to support this, but a typical test usually is 3-5 repeats assuming a normal distribution of the output, each additional test adds additional expense and provides less and less new information, Jan 29, 2020 at 15:23
• @EdM Isn't that something I wanna' find out (related to the variability)? Yes, the resistance will be affected by the gas (the technical purpose of gas sensors is that gas has the main influence on them). I would start with a single gas.
– Ben
Jan 30, 2020 at 6:14
• @Dave2e I'm quite sure that in a statistics course the lecturer once said that it is a thumb of rule to use 30 runs per parameter - or per degree of freedom - or something similar. I can't really remember. I also do not remember how that enters when having more than one parameter (60 runs or 30^2 runs?..). Hence, I assumed much more repeatings than 3-5? However, shouldn't there be a statistical background behind it? I can't really find something to dive into, combined everything I find, is that it seems to be totally arbitrary/empirical.
– Ben
Jan 30, 2020 at 6:17
• Here is a pretty good reference: isobudgets.com/how-to-perform-a-repeatability-test. From your description above, this is only the first part of many additional tests, thus the future test will fulfil the 30 sample requirement. Jan 30, 2020 at 14:28

Start by considering how precisely you need to know the variability among runs. That should be informed by the differences in sensor resistances you ultimately want to be able to detect among gases. When you eventually get to the point of comparing gases, note the following rule of thumb* for the number needed in each group ($$n$$) to detect a difference in mean values between 2 groups having normal distributions and the same variances:

$$n =\frac{16}{\Delta^2}$$

where $$\Delta$$ is the standardized difference, the ratio between the difference you would like to be able to detect $$(\delta)$$ and the standard deviation $$(s)$$ of the measurements, $$\Delta=\frac{\delta}{s}$$.

So if you only care about differences among gases on the order of 100 resistance units, it won't matter much whether the standard deviation among runs on the same gas is 0.1 unit or 1 unit or 10 units. The rule of thumb says if you only care about differences on the order of 100 units you might get away with single measurements on each group unless the standard deviation is on the order of 25 $$(\Delta = 100/25 = 4),$$ although you would probably want to do 2 or 3 regardless to rule out some catastrophic error. Note also that this rule of thumb shows you how much different values of the standard deviation would matter in terms of ultimate experimental design.

The reason for first thinking hard about the precision that you need in your variance estimate is that "learning a variance is hard," as @cardinal put it in this answer. The variance of a variance estimate when sampling from a normal distribution goes with the 4th power of the standard deviation and decreases only about linearly in the number of observations, so you need a lot of observations to pin down limits on the true value of the variance. You don't want to spend more effort on determining the variance than is necessary.

I would recommend starting with a pilot study that puts more emphasis on multiple gases and less emphasis on the number of runs per gas. Use a panel of gases likely to cover the range of expected resistance measurements and start with about 4 tests on each gas. Standard linear modeling (analysis of variance) of that type of pilot study will give you information both about likely differences of interest between gases $$(\delta)$$ and, by pooling information from among the gases, the standard deviation of measurements $$(s)$$ around the corresponding mean values. You then use diagnostic tools for linear models to examine whether assumptions like constant standard deviation among gases are reasonable, and if needed adapt your approach accordingly.

You might also want to design your pilot study in a way that can help you estimate potential carry-over among consecutive measurements on different gases. This sensor system presumably depends on gas adsorption to the resistive element, which can be quite tenacious and hard to displace. Rather than doing your tests by "waiting several hours longer than necessary" to get some idealized estimate of measurement errors, you might be better off running your tests more like they will be performed in practice and learning early on just how big a problem carry-over might be. My former life as a chemist makes me suspect that in practice that will be a larger problem than the error inherent in the resistance measurements.

*Gerald van Belle, Statistical Rules of Thumb, 2nd edition, Wiley, page 29.

• Nice informative answer (+1) Jan 31, 2020 at 19:48
• Thanks a lot! It's a bit slightly over my head resp. have to think about some details. Thus, I have some questions: Though you say the differences I wanna' observe are important I do not really know how to anser this question. Do I look for 100 Ohm differences? Or 1000 Ohm differences? When I want to evaluate them in general, isn't this a question that should be answered by the experiment/analysis?
– Ben
Feb 5, 2020 at 9:15