Estimation of measurement reliability

Question

I have the following statistical setting:

I have two unknowns $a$ and $b$ an I have two measurements of their values corrupted by noise distibuted uniformly in $[-e,e]$ and I have exact value of $e$ known to me beforehand. I also have a measurement of $d = a - b$ obtained by another method. This measurement is corrupted by a Gaussian noise with variance that I don't know in general, but I can obtain it's estimate in principle if it's necessary. Also this estimate $d$ can be subject to a gross error (outlier) with very large variance so in general the error in $d$ is given by a mixture of a Gaussian and a uniform. The mixing parameter that determines relative weight of a Gaussian and a uniform in the mixture is not known to me and cannot be obtained by any means. Also the method for measuring difference of $a$ and $b$ is way more accurate then separate estimates of $a$ and $b$ when there is no outlier in the mix.

My goal is to establish a reliability measure of measurement $d=a-b$. It's clear that individual measurements of $a$ and $b$ can in certain cases help me identify outliers in $d$. If we consider all possible values of a and b that could have resulted in the measurements that I observe, there may be no possible pair that is close to obtained measurement of $d$ and it clearly suggests that there's an outlier in such a case.

This reliability measure is supposed to be used in a big least squares estimator. I have millions of $a$'s and $b$'s and millions of measurements of their differences. I do a big least squares fit of all this data by minimizing a weighted sum of residuals given by individual measurements and pairwise difference measurements and I want to weight pairwise differences by some weight that reflects their reliability and it's really important to come up with such a measure since outliers will affect least squares alot if they are not attended to.

I see many heuristical ways to cook up such a measure, the most simple being 0 weight to difference measurements that completely disagre with observed individual measurements and 1 otherwise.

But maybe there is a better way of doing this. Do you guys see some non heuristical theoretically sound approach to divise a reliability measure that is to be used in least squares (essentially Maximum Likelihood) estimator in such a setting?

Answer 1

Your measurement process is somewhat complex. The following is the case

$ \begin{split} a &= {\cal A} + \epsilon \\ b &= {\cal B} + \epsilon \\ \epsilon &\sim {\rm u}(-e,e) \\ \\ d &= ({\cal A} - {\cal B}) + \xi \\ \xi &\sim {\rm g}(0,\sigma^2) \end{split} $

where u denotes the uniform distribution and g the normal distribution.

Hence, your noise process looks as follows

$ d_i = \left ( {\cal A} - {\cal B} \right) \circledast g(0,\sigma^2) $

with $\circledast$ the convolution operator and $i$ the index of the i'th measurement in your sample.

Robustness

You require a reliability measure for the measurement $d$. For a classification problem where $d$ has to exceed some threshold $\tau$ to be assigned class label $1$, there is a solution at hand. One such attempt is called robustness, which is defined as follows

"The robustness of a classification of a case is the probability that the case would obtain the same class label if the (unknown) true attribute values were known." [Egmont-Petersen, 1997]

In the reference, the robustness measure $\varrho$ is defined as

$ \begin{split} \varrho(d) = \int_{S} p({\cal D} \mid d) \, d{\cal D} \end{split} $

where ${\cal D}$ is the true but unknown value whereas $d$ is the observed value with all the noise in it. The region $S$ indicates the part of the measurement space (the interval) that is associated with the measured outcome $d$. The decision threshold $\tau$ demarcates the boundary of the interval $S$.

So the measure $\varrho$ integrates the conditional density $p({\cal D} \mid d)$, over the interval where ${\cal D}$ would have resulted in the same class outcome $1$.

The density $p({\cal D} \mid d)$ is unknown, but we know $p(d \mid {\cal D})$ from the convolution formula above. This convolution does have an analytical solution.

Now to robustness, we apply Bayes rule

$ \begin{split} \varrho(d) = \int_{S} \frac{p(d \mid {\cal D}) p({\cal D})}{p({\cal d})} \, d{\cal D} \end{split} $

and you have a reliability measure for your measurement $d$ - the robustness $\varrho(d)$. In practical sense, $p(d \mid {\cal D})$ is given by the convolution above. The density $p({\cal D})$ needs to be obtained from some controlled experiment with repeated measurements. Finally, $p({\cal d})$ is just the distribution of $d$ over all your data.

This picture shows the computation of $\varrho(d)$ for a given measurement $d$, for a two-class situation.

Reference

M. Egmont-Petersen, J.L. Talmon, A. Hasman1, "Robustness metrics for measuring the influence of additive noise on the performance of statistical classifiers," International Journal of Medical Informatics, Vol. 46, No. 2, pp. 103-112, 1997.