# Balancing measurement error with number of samples

Suppose I am doing a physical experiment and would like to measure the output (random variable). Inherently, I introduce measurement errors when sampling the random variable. There are also sampling errors, due to sampling only a finite number of realizations of my random variable. Is there literature relating to how to balance these two types of errors?

It is not hard to imagine a scenario where I can take more samples if I reduce how precise of a measurement device I use, and so a natural question is how do I decide what precision to use.

For instance, suppose that by rounding my measurements to the nearest centimeter rather than millimeter, I can increase the number of samples I can take by a factor of 5. Which precision should I use?

I am aware of Sheppard's corrections, but I don't think those are general enough for all cases; e.g. if my data is discrete. Moreover, even in the continuous case, Sheppard's corrections say that the measurement errors do not affect the mean. This is reasonable if you're using a relatively fine measurement system, but is clearly not true if your measurement precision is very low.

To clarify, I am considering the case where the rounding error is a deterministic function of my original random variable; i.e. assume that I sample in infinite precision, and then round to my measurement system (say the integers).

• I recommend looking up “Propagation of uncertainty” in wiki and I also recommend looking at this: QUAM:2012.P1 (EURACHEM/CITAC Guide, “Quantifying Uncertainty in Analytical Measurement”, 3rd Ed., 2012.) You can easily find it online, for free, and it is all about quantifying uncertainties in measurements, though with an obvious focus on analytical chemistry-related measurements.
– Ed V
Mar 17 '20 at 18:19
• Thanks for the response, the text looks useful. However, my understanding is that uncertainty usually is assumed to be random, while I am interested in a case where the errors are completely deterministic functions of the random variable.
– tch
Mar 18 '20 at 3:37
• Getting closer to being able to answer: model here. Scroll way down if necessary. I will run the sims starting tomorrow and see what gives. This problem turns out to be 'auto-dithering', as it were: the measurement noise dithers the quantization noise.
– Ed V
Mar 22 '20 at 0:56
• I ran the simulations and nothing unexpected happened. When the measurement noise was much smaller than the rounding error, it is simply a gross error and needs to be fixed. Vice versa is the normal way to do things and when the measurement error is commensurate in magnitude with the rounding error, it just means that the rounding error needs to be reduced. Sorry this was not useful or helpful.
– Ed V
Mar 27 '20 at 17:25

A few thoughts (BTW, I'm analytical chemist, so I think very much along the lines that @EdV suggests), and I'd like to admit first that I've never had to worry about working with such coarse measurement tools. Anyways:

### Random and other errors

my understanding is that uncertainty usually is assumed to be random

No. At least in analytical chemistry, we typically distinguish

• gross errors: Something disturbs the measurement that could be avoided (say, breaking your sample, spilling a solution, ...).
Solution: avoid (good laboratory practice), if it happens, redo the experiment.
• systematic errors (bias): can be measured and corrected, and this is typically what is done about them.: correct (to the extent that the remaining systematic error is small compared to the total uncertainty).

• random errors: get so much attention because we cannot do very much against them, we can never really get rid of them, and even reduction is a whole lot of work. The only thing we can do is: replicates.

IMHO similar to that the factor can be fixed or random depending on the task at hand, a factor may "change" its role from random to systematic (see below).

Inherently, I introduce measurement errors when sampling the random variable. There are also sampling errors, due to sampling only a finite number of realizations of my random variable.

With this, I'm not yet entirely sure I understood what exactly you are saying, but I guess from the context that the first part would be the saw-tooth-error introduced by your coarse/rounding measurement. I usually wouldn't say you introduce error by performing replicates: if we have a source of random error, a single sample would be subject to that error like any number of replicates. But the finite number of replicates (of this factor) limits how well you can estimate e.g. the mean of that distribution.

### How to balance errors/ sources of uncertainty?

For sources of systematic error, measuring the systematic error and correcting for it is typically the approach of choice. After the correction you typically have some random "leftovers" of an imperfect correction since the systematic deviation was measured and the correction is thus itself subject to random error.

For independent sources of random error it is usually most efficient to balance replicates for each of them so that in the end each of these sources of noise contributes roughly the same amount of variance to your final result.

The reasoning behind this that the largest variance dominates total uncertainty, so putting more effort into replication for that source of error helps, while the same effort put into a minor contributor of random error won't have a substantial effect the total uncertainty.

In practice, this is often adjusted a bit according to how costly replicates are for which source of error. If for one particular factor replicates are very cheap, one may as well decide to do sufficient replicates so that this factor becomes small or negligible.

If you look at total uncertainty as opposed to total variance, systematic error can be treated just the same: the guiding principle is: how much effort (and where) "buys" how much improvement for the total uncertainty?

Random error can beceome systematic error further along the calculations. E.g. random error on a calibration measurement becomes systematic for all measurements that use this calibration. In such a situation, one better replicate so that the resulting uncertainty is negligible among the total uncertainty for the measurement that is to be calibrated.

There are rare occasions where at least a rough balancing can be done by back-of-the-envelope calculations before the experiments start (e.g. Poisson noise such as optical shot noise, or measuring proportions) but in general the balancing needs preliminary estimates of the relevant sources of error/uncertainty.

## Binning / coarse measurement tools

Sheppard's corrections say that the measurement errors do not affect the mean. This is reasonable if you're using a relatively fine measurement system, but is clearly not true if your measurement precision is very low.

Garbage in - garbage out is true also in statistics: statistics are no magic bullet against bad measurements. Binning removes information, and at some point so few information is left that nothing can be said any more. If the mean is affected by your binning, this is a sign that the measurement is too coarse and cannot be used (in analytical chemical language: is not fit for purpose).

So the question is roughly: what's the small print of Sheppard's Corrections?

A quick search yielded 3 promising looking papers:

(I've only glanced over the 2004 paper and don't have access to the other 2)

Seeing @EdV's "non-answer", maybe we can approach the problem the other way round: bin width h can "hide" a variance of 0.25 h², thus a standard deviation of 0.5 h. Hide in the sense that this is the largest possible variance that may not be detected at all, and that could happen in a way where no whatsoever replication helps. So I'd argue for now that unless we can make "nicer" assumptions about the distribution of the measured random variable, I'd say there's a squared error of up to 1/4 h² caused by the discretization procedure. And that's something we can relate to other sources of uncertainty (and for which we can also do error propagaton) and decide whether that is negligible or not negligible or overwhelming.

Measuring 600 coins won't help because that replicates the wrong influencing factor.
In an analytical-chemical analogy, this is similar to a situation where I have a field sampling error of 100 and instrument noise of 1: doing 100 replicate measurements with the instrument will not help at all.

• It finally dawned on me that this problem is basically "How many bits resolution does an analog-to-digital converter (ADC) need to have in order to not lower the signal-to-noise ratio (SNR) of its noisy input?" Back in the 1990's, I taught some grad classes where I looked at this, so I dug up my computer simulations, cleaned them up a little bit and this shows some screenshots of one of the simulation models. What is at the link is not an answer, but now I think I can do something useful with this problem. More later.
– Ed V
Mar 21 '20 at 2:35