# Statistical Tests That Incorporate Measurement Uncertainty

Suppose I am given two groups of mass measurements (in mg), which are referred to as y1 and y2. I want to do a test to determine if the two samples are drawn from populations with different means. Something like this for example (in R):

y1 <- c(10.5,2.9,2.0,4.4,2.8,5.9,4.2,2.7,4.7,6.6)
y2 <- c(3.8,4.3,2.8,5.0,9.3,6.0,7.6,3.8,6.8,7.9)
t.test(y1,y2)


I get a p-value of 0.3234, and at a significance level of 0.05 don't reject the null hypothesis that the two groups are drawn from populations with the same mean. Now I am given uncertainties for each measurement:

u1 <- c(2.3,1.7,1.7,1.7,2.0,2.2,2.1,1.7,2.3,2.2)
u2 <- c(2.4,1.8,1.6,2.3,2.5,1.8,1.9,1.5,2.3,2.3)


where u1[1] is the combined standard uncertainty in measurement y1[1] (and so on). How do I incorporate these uncertainties into the statistical test?

• Are these paired measurements or not? (I assume not.) The uncertainties could be used derive weights which could improve your inference, but the variation in uncertainties is pretty small, so there's not going to be much gain, even in the uncertainties are accurate. May 8, 2013 at 2:28
• These are a subset of real unpaired data. The example was primarily intended to clarify the question. What I am really looking for is general guidance on how to best incorporate measurement uncertainty into a hypothesis test (like a t test). It seems to me that we are wasting a lot of valuable information if we don't use the measurement uncertainties, but I have been unable to find clear guidance on this subject in the literature.
– Tom
May 8, 2013 at 10:28
• To make fullest use of them you need to incorporate them into a probabilistic model for the observations; what do the measurement uncertainties actually represent? (You can't handwave this, so be careful.) May 8, 2013 at 10:44

It sounds like you want to conduct a weighted analysis. See the "Weighted Statistics Example" in the "Concepts" section of the SAS documentation.

• So we just do the t-test with the weighted means and weighted standard deviations, where u1 and u1 are the weights?
– Tom
May 8, 2013 at 19:24
• Yes. You would be assuming that the variance of the i_th observation is Var/w_i, where w_i is the weight for the i_th observation and Var>0.
– Rick
May 16, 2013 at 12:46

Why not simulate it? That is, add in your uncertainty as realizations of noise to each observation. Then repeat the hypothesis test. Do this about 1000 times and see how many times the null was rejected. You will need to pick a distribution for the noise. The normal seems like one option, but it could produce negative observations, which is not realistic.

You could turn it into a regression problem and use the uncertainties as weights. That is, predict group (1 or 2?) from measurement in a regression.

But

The uncertainties are approximately constant, so it seems likely that nothing much will change by using them too.

You have a mild outlier at 10.5, which is complicating matters by reducing the difference between means. But if you can believe the uncertainties, that value is no more suspect than any others.

The t-test does not know that your alternative hypothesis is that two samples are drawn from different populations. All it knows about is comparing means, under certain assumptions. Rank-based tests are an alternative, but if you are interested in these data as measurements, they don't sound preferable for your goals.

• Point taken. I changed the question to express it in terms of the means.
– Tom
May 7, 2013 at 16:34

In ordinary least squares (e.g., lm(y~x)) you are allowing for variability (uncertainty) around y values, given an x value. If you flip the regression around (lm(x~)) you minimize the errors around x. In both cases, the errors are assume to be fairly homogenous.

If you know the amount of variance around each observation of your response variable, and that variance is not constant when ordered by x, then you would want to use weighted least squares. You can weight the y values by factors of 1/(variance).

In the case where you are concerned that both x and y have uncertainty, and that the uncertainty is not the same between the two, then you don't want to simply minimize residuals (address uncertainty) in perpendicular to one of your axes. Ideally, you would minimize uncertainty that is perpendicular to the fitted trend line. To do this, you could use PCA regression (also known as orthogonal regression, or total least squares. There are R packages for PCA regression, and there have previously been posts on this topic on this web site, which have then also been discussed elsewhere. Furthermore, I think (i.e., I may be wrong...) you can still do a weighted version of this regression, making use of your knowledge of the variances.