2
$\begingroup$

The isotopic analyses of two tissues across 50 specimens showed a mean difference of 0.12 ‰. A Wilcoxon signed-rank test for paired samples indicated this to be statistically significant (Z: -2.515, P = 0.012).

However, analytical uncertainty (based on the replicate analysis of standards) was calculated to be ± 0.18 ‰. As this is greater than the mean difference, are the statistical results still valid?

Is there a way of taking this uncertainty into account? Or an alternative analysis which should be undertaken?

Thanks in advance.

EDIT: Thank you very much for all the comments and answers provided so far, I am very grateful. The components of variance calculations provided by whuber are exactly was I was looking for. Thanks again.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Difficult to say without knowing the meaning of and basis for the 'analytic uncertainty' value you quote. If it refers to individual observations, then I don't see a basis for comparing that with a CI based on a sample with many observations. Will show an example in Answer. $\endgroup$
    – BruceET
    Commented May 6, 2020 at 16:21
  • $\begingroup$ What is 'analytical uncertainty' and what is 'replicate analysis of standards'? Did you do something like repeated measurements on the same specimens? But what did you do and compute exactly? $\endgroup$
    – Sextus Empiricus
    Commented May 6, 2020 at 21:31
  • 1
    $\begingroup$ Relating to BruceET's answer. If 0.18 is the estimated standard deviation of a single measurement, then 0.025 is the standard deviation for the mean of 50 measurements, and 0.025 x sqrt(2) ~ 0.036 will be the standard deviation for the difference between the means of two such measurements. Your difference of 0.12 is 3.3 times larger than this 0.036, so on that aspect you do not need to worry much. (Note that this 3.3 is larger than the z-value, this means that the variation of the observations is larger than measurement error and possibly also due the distribution of the weights) $\endgroup$
    – Sextus Empiricus
    Commented May 6, 2020 at 21:48
  • $\begingroup$ Plotting a histogram for the weights of both tissues might be helpful to get an intuitive idea. $\endgroup$
    – Sextus Empiricus
    Commented May 6, 2020 at 21:55

2 Answers 2

Reset to default
5
$\begingroup$

Suppose you are trying to weigh a package on a scale that gives unbiased readings but is subject to variations from one weighing to the next.

If the true weight of the box is 960g and we have the patience to use this scale to weigh the package 25 times. Then the 25 results might be as follows.

set.seed(2020)
x = round(rnorm(25, 995, 5))
x
 [1]  997  997  990  989  981  999 1000  994 1004  996
[11]  991 1000 1001  993  994 1004 1004  980  984  995
[21] 1006 1000  997  995  999
summary(x); sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  980.0   993.0   997.0   995.6  1000.0  1006.0 
[1] 6.879922

Our measurements span an interval from 980 to 1006g and the sample standard deviation is about 6.88g. I don't know what its 'analytic uncertainty' would be. But I would feel comfortable putting postage on it for a package 'up to one kg.'---hoping that the post office has better scales than mine, if they decide to verify the weight.

A one-sided 95% confidence interval for the weight of the box has an upper limit of about 998g.

t.test(x, mu=1000, alt="less")

        One Sample t-test

data:  x
t = -3.1977, df = 24, p-value = 0.001931
alternative hypothesis: true mean is less than 1000
95 percent confidence interval:
     -Inf 997.9541
sample estimates:
mean of x 
    995.6

A two-sided 95% confidence interval is $(992,76, 998.44)$ or $995.6 \pm 2.84,$ so the 95% margin of error is $2.84.$

t.test(x)$conf.int
[1] 992.7601 998.4399
attr(,"conf.level")
[1] 0.95

Addendum: In the figure below, the black curve is the density curve for the population of weight measurements, which is $\mathsf{Norm}(\mu = 995, \sigma = 5).$ That determines the variability of individual measurements $X_i.$

The blue curve is the density curve for $\bar X,$ means of samples of size $n=25.$ Its standard deviation is $\sigma_{\bar X} = \sigma/\sqrt{n} = 5/\sqrt{25} = 1.$ This curve governs the margin of error of a 95% CI based on 25 observations. It is one fifth as 'wide' as the population density and so five times as 'tall'. Both curves enclose a total probability of $1.$

enter image description here

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thank you very much for the thorough response! To continue the weighing scales analogy: these scales showed a 95% margin of error of 2.84 from an object with a known weight. We then used these scales to weigh parcels destined for France and Germany of an unknown weight. Those for Germany were on average 2.33g heavier, which a statistical test determined to be significant. Does that hold true when we know the scales have an error margin of 2.84? Perhaps as the magnitude of error is equal in each group, and randomly distributed, is this already built into the statistical comparison? $\endgroup$
    – Sean Paul
    Commented May 6, 2020 at 18:25
  • 1
    $\begingroup$ The weight of my pkg was 'known' only because provided simulated data. // The point is that with enough measurements on a scale of limited precision I can make the margin of error of the CI as small as I want. With 100 weighings the ME might go down to $\pm 1.4$g. The 'analytic uncertainty' of any one measurement does not limit the margin of error of a CI. // I assumed my scale has SD $\sigma = 5$g for each individual measurement. $\endgroup$
    – BruceET
    Commented May 6, 2020 at 19:24
  • 2
    $\begingroup$ Not only is that built into the comparison, Sean Paul, you can actually back out a component of variance. Since $Z\approx -2.5,$ we know one standard error of the mean difference is $0.12/2.5=0.048$ with a variance of $0.048^2,$ whereas the measurement error contributes $2\times 0.18^2/50$ to that variance. Subtracting gives a variance component of size $0.001$ that reflects real variation among the tissue concentrations, after accounting for the measurement error. One thing demonstrated by this result is that there is detectable variation among samples despite the uncertainty. $\endgroup$
    – whuber ♦
    Commented May 6, 2020 at 21:27
  • $\begingroup$ Thank you for all of the comments and answers, I am very grateful. The calculation offered by whuber is exactly the what I was after, and I feel more confident that the variation seen between tissues is real. $\endgroup$
    – Sean Paul
    Commented May 7, 2020 at 14:48
1
$\begingroup$

I take "analytic uncertainty" to have the same meaning as "uncertainty" in metrology--quantified doubt about a measurand. Sampling variance contributes to uncertainty but is likely not the only source. For example, if a scale is accurate under certain laboratory conditions, how confident are you that those conditions held when the data were collected? The thermometer in the lab has its own uncertainty, which now becomes part of the overall uncertainty, in addition to the sampling variance. Metrologists embrace both statistical and nonstatistical methods for quantifying doubt. So your result could be "statistically significant" yet still lie within the range of expanded uncertainty. Refer to a metrology manual for your field to determine how to proceed.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.