Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I'm currently trying to compare the measurement accuracy of two devices. However, I am using one of the devices as a "gold standard" to say something about the accuracy of the other device.

The quantity I am trying to measure is the distance (depth) to a point in a scene (image). I am using a laser scanner which gives me (x,y,z) coordinates for a point and I am using optical geometry to give me (x,y,z) coordinates for the same point. My base (or reference) measurement is the laser scanner measurement. I am using this as a gold standard. I now have a corresponding measurement from the optical geometry based measurement system. I have corresponding measurements for 1000's of points. I average the error in the difference between these and get a mean error. This error, however, neglects the fact that the laser scanner too is a measurement system and has its own error. It comes with an accuracy of +/- 5mm for an object at 10m range. How do I incorporate this into an accuracy metric for the optical geometry based system?

share|improve this question

1 Answer

up vote 1 down vote accepted

When measuring the accuracy of a measurement system you are always faced with the problem that you can't know the "truth" exactly and so have to compare to a standard. As you say your estimate of the accuracy is off because you are unable to take into account the accuracy of the standard. Hopefully you used the the method as a gold standard because it is highly accurate. If you take repeated measurements with the standard you could then get an estimate of its accuracy. For the moment assume both instrument provide unbiased estimates. Then MSE=variance and Var(X-Y) =Var(X)+ Var(Y) where X is the new measurement and Y is the standard. You have an estimate of Var(X-Y) which you have been using as a rough estimate for Var(X) now by repeated measurements or some other means you have an estimate of the variance of the standard. Subtraction the estimate of Var(Y) from the estimate of Var(X-Y) and you will have your estimate of Var(X).

share|improve this answer
I see, so you are saying Var(X-Y) = $\frac{\sum_{i=1}^{N} (X_i - Y_i)^2}{N}$ ? And that Var(X) = Var(X-Y) - Var(Y)? – Mustafa Jul 3 '12 at 20:44
Yes because I am assuming the random components of the two measurement system estimates are statistically independent and hence Cov(X,Y)=0. – Michael Chernick Jul 3 '12 at 20:48
OK, I get that, thanks. Maybe a related question I should be asking is that what is a good metric to use to comment on the accuracy of a measurement? Let's say I am measuring length. Let Y be the truth and X be the measurement. Var(X-Y) tells me the variability of my measurement about the mean. Does that mean that the E[X-Y] (expected value) should be close to zero. Intuitively, if it is non-zero, then it means that there is a bias to the measurement right? E[X] = E[X-Y] + E[Y] right? – Mustafa Jul 3 '12 at 20:56
@Mustafa That would be yes. – Michael Chernick Jul 3 '12 at 21:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.