Suppose there are two independent measurements, $x_1 \pm \sigma_1$ and $x_2 \pm \sigma_2$, both of which purport to measure the same underlying quantity $X$ (which is apriori unknown). For simplicity, we can take these to be Gaussian distributed measurements (but I'd like to be a bit more general if possible).

In the frequentist framework, how does one assess the consistency (or otherwise) of these measurements? That is, how does one make quantitative statements about whether the data are indeed consistent with measuring the same quantity $X$? Of course, intuitively, for $x_1$ and $x_2$ which are far apart (relative to something like $\sqrt{\sigma_1^2 + \sigma_2^2}$), we would like to say they become less and less consistent.

I understand how to approach this problem from a Bayesian perspective: evaluate the Bayesian evidence for two models, one in which the measurements are derived from the same underlying quantity $X$, and one in which they are generated by different underlying quantities ($X_1$ and $X_2$, say). Of course this requires specification of priors on $X$, $X_1$ and $X_2$ (and also priors on the two models).

Attempted solution

Define a null hypothesis in which they do indeed come from the same underlying quantity $X$. Calculate the probability $p(X) = p_1(X)p_2(X)$ of observing 'more extreme' measurements under this model. This can be done analytically for Gaussian distributed measurements, yielding a product of two complementary error functions. The answer is a function of the true (unknown apriori) $X$.

Should one then pick the maximum likelihood $\hat{X}$, plug this into $p(\hat{X})$ and compare that p-value to some threshold $\alpha = 0.05$ (say)? This $\hat{X}$ is in general different from the $X$ which maximises $p(X)$.



You are almost there.

Say we are measuring the observable $X$ with the accuracy given by variance $\sigma_1^2$. If the true value of $X$ is $\mu$, then the measured value should be random variable whose probability density function (PDF) is given by the Gaussian distribution $N(\mu,\sigma_1)$. Since we do not know the true value of $X$, the best thing we can give as the result of the measurement is to say: the most probable value of $X$ is the one we measure (call it $\mu_1$), the error of the measurement is $\sigma_1$ and, hence, the true value of $X$ is described with the probability density function $N(\mu_1,\sigma_1)$.

So far I said nothing special. Now, let's consider two independent estimates of the same variable $X$. If the errors of the two measurements are $\sigma_1$ and $\sigma_2$, then the measured values $\mu_1$ and $\mu_2$ are random variables given with PDFs $N(\mu,\sigma_1)$ and $N(\mu,\sigma_2)$. Here you can have any other PDF depending on the measurements, procedures and methods used for the estimates. I am taking the normal distributions as an example. We do not know the true value $\mu$, but we can calculate the probability for an absolute difference of the two measured values. Let's call it $y=x_1-x_2$, where $x_1$ and $x_2$ are the two possible values of $\mu_1$ and $\mu_2$. Then the probability for any value of $y$ results to


In the case of the Gaussian variables we get $p(y)=N\left(0,\sqrt{\sigma_1^2+\sigma_2^2}\right)$.

This result does not depend on $\mu$ and you can easily calculate the probability that the two measurements differ by the value $|\mu_1-\mu_2|$ or higher, which is the integral of the tails of $p(y)$ distribution. If this probability is lower than 5% (or 1%) you can say that the two measurements disagree or are in tension. Usually, however, the tension is expressed in numbers of $\sigma$, which is simply the ratio $(\mu_1-\mu_2)\left/\sqrt{\sigma_1^2+\sigma_2^2}\right.$. It is simply telling you the position of the value $\mu_1-\mu_2$ on the distribution of this derived observable. If this number is smaller than, say, 1.7, I would not even call it a tension. Otherwise, you can say that, e.g. the estimates are in $2\sigma$ tension, or $2.3\sigma$ away from each other.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.