Validate a medical test

Question

I've never done something like this before, thus I do not even know where to start with calculating whatever I need.

Following scenario:

We measure a medical value (Glucose) with our device, and also measure a laboratory reference value to compare our value to.

Now for a paper I was asked to deliver some sort of value which specifies how good our measurements are.

I know of correlation as a measurement of how dependent of each other two measurements are, but I have no idea what sort of value is being asked for in this sort of tests?

I read something about Level of Confidence, which sounds good, but I don't understand how I would apply it here.

The data I have is (basically) two arrays, each with 30 floating values. What do I do from here?

Answer 1

You are doing a (chemical) calibration, and the search phrase you are looking for is method validation in analytical chemistry.

• There actually exist norms how to validate methods in analytical chemistry, and certain measures of performance like limit of detection (LOD), limit of quantitation (LOQ, probably more relevant for you), recovery rate, etc.

• If you can read German, the Wiki page about method validation is a decent starting point (unfortunately, there is no English version).

• I like Handbuch Validierung in der Analytik but I'm not aware of an English translation.

• Have a look through the analytical chemistry section of your library (if that exists).

• And you probably should look up the difference between the confidence interval of a calibration and prediction intervals.

• We won't be able to give you any more detailed advise without knowing how you arrive at your concentration: some of the "standard" techniques to calculate e.g. the LOD, or the confidence and prediction intervals in linear calibration are valid only for ordinary least squares (which in chemometrics is usually suitable only for univariate calibration).
  See e.g.

  • Klaas Faber and Bruce R. Kowalski: Propagation of measurement errors for the validation of predictions obtained by principal component regression and partial least squares, Journal of Chemometrics, 11, Issue 3, 181–238, 1997.

  • We observed quite different LODs for the "blank" method vs. the "relative error" method here: S. Dochow et al.: Raman-on-chip device and detection fibres with fibre Bragg grating for analysis of solutions and particles. Lab Chip, 2013, 13, 1109-1113.

  • One common difficulty is that the usual methods for calculating confidence (and prediction) intervals assume all calibration samples to be independent. But this is frequently not the case, e.g. when calibration samples are prepared by diluting stock solutions (unless for each calibration sample a new stock is prepared). However, the problems can be avoided if validation is done with a distinct series of samples with is prepared from a new stock.
    The univariate methods usually calculate these intervals from within the training data. But the more sophisticated multivariate methods like principal component regression or partial least squares regression do a rotation/projection of the data first and then calculate the regression in scores space. Confidence intervals need to take into account that this projection is also derived from the calibration data, and thus has uncertainty. This source of variance is not covered by the analytical solutions for confidence intervals for ordinary least squares.

• In any case, what you can do is: bootstrap/cross validate your calibration (with respect to stock solutions/independent samples) and derive confidence intervals, RMSE and (relative) error and from that the figures of merit you need.

• However, if you do a univariate calibration, work in R and all measurements are from independent samples, package chemCal will give you the figures of merit and calibration plots.

Answer 2

Your statement 'also measure a laboratory reference value to compare our value to' is slightly ambiguous. I assume you mean 'get Glucose measures from the same samples assessed by a lab'... (but you might mean that the second sample are your device's assessment of known glucose-level sample that aren't related to the first lot of values).

If the laboratory reference values are a gold standard, wouldn't you want to know two things:

1) typical bias

2) some kind of typical distance from the standard, like average absolute percentage error or RMSE...?

and for a more sophisticated idea of how your machine performs:

3) some kind of plot of error (or %error) vs gold standard (to check for a trend)

However, since you have both your measurements and the gold standard, you can also calibrate your measurements to the standard, which might eliminate the bias (1) almost completely and reduce (2) to pure variability. That is, imagine the correlation is really high but (1) and (2) look really bad. You should be able to use regression to derive a linear correction that greatly improves the absolute accuracy of your results.

More algebraically: if your device values are $y_i$, $i = 1, 2, ..., n$, and the gold standard values are $m_i$, wouldn't you want

1) average of $y_i - m_i$ OR $(y_i - m_i)/m_i *100\%$

2) average of $|y_i - m_i|$ OR $|y_i - m_i|/m_i *100\%$ OR RMS(y_i - m_i) or RMS((y_i - m_i)/m_i)

3) plot of $y_i - m_i$ OR $(y_i - m_i)/m_i *100\%$ vs $m_i$

More than likely, I bet there's some typical measure like one of those in (1) and (2) that are widely used for devices like yours already ... and you should definitely use whatever that is.

See also

Calibration curve