# Errors-in-variables regression: is it valid to pool data from three sites?

I recently had a client come to me to do a bootstrap analysis because an FDA reviewer said that their errors-in-variables regression was invalid because when pooling data from sites the analysis include pooling data from three sites where two sites included some samples that were the same.

BACKGROUND

The client had a new assaying method they wanted to show was "equivalent" to an existing approved method. Their approach was to compare the results of both methods applied to the same samples. Three sites were used to do the testing. Errors-in-variables (Deming regression) was applied to the data at each site. The idea is that if the regression showed the slope parameter to be close to 1 and the intercept near 0 this would show that the two assaying techniques gave nearly the same results and hence the new method should be approved. At site 1 they had 45 samples giving them 45 paired observations. Site 2 had 40 samples and site 3, 43 samples. They did three separate Deming regressions (assuming a ratio of 1 for the measurement errors for the two methods). So the algorithm minimized the sum of squared perpendicular distances. Separate regressions were done at each site and a pooled regression was also done using all the data from all three sites.

In their submission the client pointed out that some of the samples used at sites 1 and 2 were the same. In the review the FDA reviewer said that the Deming regression was invalid because common samples were used which causes "interference" that invalidates the assumptions of the model. They requested that a bootstrap adjustment be applied to the Deming results to take account of this interference.

At that point since the client did not know how to do the bootstrap I was brought in. The term interference was strange and I was not sure exactly what the reviewer was getting at. I assumed that the point really was that the because the pooled data had common samples there would be correlation for the common samples and hence the model error terms would not all be independent.

THE CLIENT'S ANALYSIS

The three separate regressions were very similar. Each had slope parameters close to 1 and intercepts near 0. The 95% confidence interval contained 1 and 0 for the slope and intercept respectively in each case. The main difference was a slightly higher residual variance at site 3. Furthermore they compared this to the results from doing OLS and found them to be very similar (in only one case did the confidence interval for the slope based on OLS not contain 1). In the case where the OLS CI for the slope did not contain 1 the upper bound of the interval was something like 0.99.

With the results being so similar at all three sites pooling the site data seemed reasonable. The client did a pooled Deming regression which also lead to similar results. Given these results I wrote a report for the client disputing the claim that the regressions were invalid. My argument is that because there are similar measurement errors in both variables the client was right to use Deming regression as a way to show agreement / disagreement. The individual site regressions had no problems of correlated errors because no samples were repeated within a given site. Pooling data to get tighter confidence intervals. The pooling which included use of the the common samples twice might produce positive correlation between the residuals for those common samples which would mean that the confidence intervals for the regression parameters would be too narrow (the estimated model residual variance biased on the low side).

This difficulty could be remedied by simply pooling the data with the common samples from site 1 say left out. Also the three individual site models do not have the problem and are valid. This seems to me to provide strong evidence of agreement even without the pooling. Furthermore the measurements were taken independently at sites 1 and 2 for the common sites. So I think that even the pooled analysis using all the data is valid because the measurement errors for a sample at site 1 are not correlated with the measurement errors in the corresponding sample at site 2. This really just amounts to repeating a point in the design space which should not be a problem. It does not create correlation / "interference".

In my report I wrote that a bootstrap analysis was unnecessary because there is no correlation to adjust for. The three site models were valid (no possible "interference" within sites) and a pooled analysis could be done removing the common samples at site 1 when doing the pooling. Such a pooled analysis could not have an interference problem. A bootstrap adjustment would not be necessary because there is no bias to adjust for.

CONCLUSION

The client agreed with my analysis but was afraid to take it to the FDA. They want me to do the bootstrap adjustment anyway.

MY QUESTIONS

A) Do you agree with (1) My analysis of the client's results and (2) my argument that the bootstrap is unnecessary.

B) Given that I have to bootstrap the Deming regression are there any procedures SAS or R that are available for me to do the Deming regression on the bootstrap samples?

EDIT: Given the suggestion of Bill Huber I plan to look at bounds on the errors-in-variables regression by regression both y on x and x on y. We already know that for one version of OLS the answer is essentially the same as errors-in-variables when the two error variances are assumed to be equal. If this is true for the other regression then I think that will show that the Deming regression gives an appropriate solution. Do you agree?

In order to meet the client's request I need to do the requested bootstrap analysis that was vaguely defined. Ethically I think it would be wrong to just provide the bootstrap because it doesn't really solve the client's real problem, which is to justify their assay measuring procedure. So I will give them both analyses and request at least that they tell the FDA that in addition to do the bootstrap I did inverse regression and bounded the Deming regressions which I think is more appropriate. Also I think that analysis will show that their method is equivalent to the reference and the Deming regression is therefore adequate also.

I plan to use the R program that @whuber suggested in his answer to enable me to bootstrap the Deming regression. I am not very familiar with R but I think I can do it. I have R installed along with R Studio. Will that make it easy enough for a novice like me?

Also I have SAS and am more comfortable programming in SAS. So if anyone knows a way to do this in SAS I would appreciate knowing about it.

-
I don't know the answer to this question, but, on a purely political basis, wouldn't it be better to do what the FDA wants and show (at least, presumably), that the results are similar? (Good question, BTW, +1) – Peter Flom Sep 26 '12 at 11:21
Yes @PeterFlom I agree that doing the analysis for the FDA and showing it doesn't matter. But I think that diplomatically pointing out the results of the regressions and their implications and doing the pooling without the overlapping samples strengthens the argument. I am going to do the bootstrap but I could use help finding available software to do the Deming regression myself without independently coding it up. – Michael Chernick Sep 26 '12 at 11:40
Michael, the possibility of "samples" common to "sites" calls into question some natural interpretations of what these (abstract) terms might mean. For instance, I initially thought of "sites" as different geographic locations and "samples" as separate entities associated with those locations, each subjected to independent measurements. In this model it is impossible for samples to be common to different sites. Could you please clarify what you mean by these terms? – whuber Sep 26 '12 at 13:16
@whuber the sites are different locations. The samples are citrated plasma from individuals. The lab testing is done at the different sites at different times. The comparisons are for two assay measuring devices that are intended to do the same function. At sites 1 and 2 some of the samples were reused but the devices operated independently at site 1 and site 2. So that is why I say the measurement errors are really independent even though the same samples (or portions of the same samples) are used. – Michael Chernick Sep 26 '12 at 13:58
a) Agreed that leaving out the duplicated sample from the pooled analysis removes the concerns about lack of independence. b) Very few SAS users are going find it "easy" to use R for bootstrap analyses involving uncommon regression methods. Bootstrap analyses really do require the functional programming mode of thinking, and that is not a mode that SAS encourages. – DWin Sep 29 '12 at 15:49

This is a mutual calibration problem: that is, of quantitatively comparing two independent measurement devices.

There appear to be two principal issues. The first (which is only implicit in the question) is in framing the problem: how should one determine whether a new method is "equivalent" to an approved one? The second concerns how to analyze data in which some samples may have been measured more than once.

### Framing the question

The best (and perhaps obvious) solution to the stated problem is to evaluate the new method using samples with accurately known values obtained from comparable media (such as human plasma). (This is usually done by spiking actual samples with standard materials of known concentration.) Because this has not been done, let's assume it is either not possible or would not be acceptable to the regulators (for whatever reason). Thus, we are reduced to comparing two measurement methods, one of which is being used as a reference because it is believed to be accurate and reproducible (but without perfect precision).

In effect, the client will be requesting that the FDA allow the new method as a proxy or surrogate for the approved method. As such, their burden is to demonstrate that results from the new method will predict, with sufficient accuracy, what the approved method would have determined had it been applied. The subtle aspect of this is that we are not attempting to predict the true values themselves--we don't even know them. Thus, errors-in-variables regression might not be the most appropriate way to analyze these data.

The usual solution in such cases is "inverse regression" (as described, for instance, in Draper & Smith, Applied Regression Analysis (Second Edition), section 1.7). Briefly, this technique regresses the new method's results $Y$ against the approved method's results $X$, erects a suitable prediction interval, and then functionally inverts that interval to obtain ranges of $X$ for any given values of $Y$. If, for the intended range of $Y$ values, these ranges of $X$ are "sufficiently small," then $Y$ is an effective proxy for $X$. (In my experience this approach tends to be conservatively stringent: these intervals can be surprisingly large unless both measurements are highly accurate, precise, and linearly related.)