# Linear least-square fitting of two variables with uncertainty on both

I am trying to find an R function to calculate the linear least-square fitting of two variables when both have an error (expressed as standard deviation). I have found this problem referred to in half a dozen different ways: Williamson-York method, RMA, Deming regression, bivariate fit, weighted least squares, etc...

I am no statistician, so this may seem like a stupid question, but are all these names referring to the same methodology? And what is the correct name for it if one needs to look for information?

I need to use the procedure described in this paper: https://acp.copernicus.org/articles/8/5477/2008/acp-8-5477-2008.pdf. The assumptions are that the distribution of errors is normal and and the fit parameters do not depend on the choice of units. The R deming package seems to give the same results as in the paper, at least on the example dataset, so I think it is okay. But it does not return a Pearson correlation coefficient, like lm does.

Is it that this type of regression cannot calculate the correlation coefficient or is it just the particular implementation of the deming package that does not?

EDIT: in a related paper, I found reference to this C function http://numerical.recipes/webnotes/nr3web19.pdf. It seems to be doing what I want but it is unclear to me if it is the same fitting method.

EDIT 2: specific questions:

1. Are these terms referring to the same method: Williamson-York method, RMA, Deming regression, bivariate fit, weighted least squares?
2. It the methods are equivalent, which one is the correct term to use?
3. Are the methods described in the two linked pdf the same?
4. The R package deming implements a Deming regression. Why does it not return a correlation coefficient? Is it the method or its implementation in the package?
• One algorithm which suits your task is named 'total least squares' or 'Gauss-Helmert Model'. They imply that both variables are uncertain might have different variances. – nali Dec 30 '20 at 20:35
• What is your research question ? If you just want a correlation coefficient, then you can compute it easily from the 2 variables; otherwise deming regression sounds like exactly what you need. – Robert Long Mar 3 at 14:00
• My specific application is the calibration of an instrument. So I need the correct slope (calculated taking into account the uncertainties in both the x and y axis) and the associated correlation coefficient. In a more general sense I am trying to understand if all the methods mentioned here are equivalent and if not what are the differences and which is the best to use (esp wrt to the implementation in R). – rs028 Mar 3 at 15:08

First of all, I think there is a misunderstanding of what linear means in linear least squares. Linear means that it is linear in the coefficients, not necessarily in the model. A straight line fit means that it is a linear model, that happens to be linear in the coefficients as well in the standard formulation. $$f(x) = ae^{-x} + b$$ is linear in the coefficient, so one could use the linear least squares method, while the following isn't: $$f(x) = \frac{1}{a} x + \sqrt{b}$$

On to the question: It's a bit commonplace to state that one needs a certain level of expertise in the field that one is drawing methodologies from. Statistics in general is no exception; but statistical methods are in itself so abundant in any kind of empirical science (natural sciences, social sciences,...). This said, a lot of fields have specialized in certain parts of statistics, and more specifically in the applied parts. A study in medicine or psychology needs to address different issues than chemistry or physics.

So -- is there a even a correct name for anything? Yes, sometimes there is a generally accepted and understood name. But sometimes different names mean slightly different things. So, it in a particular field a certain method might have an accepted name -- and someone from a different field might not recognize it as such.

This is not about avoiding the question "is there a correct name?". Most people would call it actually just a specific case of linear least squares. In the comments already total least squares was mentioned. In physics (my field) this would be called generalized least squares.

About the choice of methods: Williamson-York, Deming regression and the numerical recipe are limited to the specific choice of a straight line fit. They are a special case of the general methods like GLS (generalized LS) or OLS (ordinary LS).

(1) Names: Most of the names that are mentioned mean something similar and related. The specifics of most the terms can be easily looked up; even on Wikipedia (e.g. here). It is important to read the publications carefully. The paper that is linked has a very specific set of requirements, in the sense that it is not a universal solution of the general case. Bivariate fit is not a unique name, it simply means that there are two variates. Deming regression would be a bivariate linear fit. RMA is not a term I'm familiar with. Weighted least squares is an alternative name for a general least squares fit.

(2) They are not the same. It depends on what the actual task is.

(3) Maybe. The code is public, so one could compare it. Both are about fitting straight lines with uncertainties. If both are correct, then both are equivalent. This is possible a take away point: results may be the same, but sometimes implementations have their own name.

(4) The standard Deming regression implementation (see here) doesn't compute the (Pearson) correlation coefficient. Questions about a particular programming package are better suited at a different Exchange site.

• RMA stands for "reduced major axis regression" – rs028 Mar 3 at 19:38
• Thanks @cherub. I would agree that each application may have different needs and therefore there is no general answer to the question "which method is better?". However the problem I am trying to solve here comes before that question, and it is to first understand the various options (which is even more complicated by the fact there appears to be no standardized nomenclature). – rs028 Mar 3 at 20:38
• @rs028 I'd like to answer your question completely. But from this comment, I'm not sure. There are three different parts that you need. You need some data, a model and method to extract the parameters from a model. You already know that you data depends on two variates, which have an associated uncertainty. I still don't know what kind of model you're trying to fit; usually this is some kind of function. But it seems that you already have settled on a parameter estimation method; namely (linear?) least squares. This "triad" of data, model and method has to fit together. – cherub Mar 3 at 21:46
• @rs028 There is a need to understand what are you trying to fit. If you try fit a straight line, then Deming regression or Williamson-York will take the uncertainties correctly into account. Which wouldn't be accounted for, if you'd use straightforward/simple "linear regression". You could also use any program/library that provides a least squares fit that accounts properly for the uncertainties. If this is called "weighted least squares", "generalized least squares" or any other name -- the results will/should be the same. It will always be some "best" values with uncertainties. – cherub Mar 3 at 22:01
• my particular application is the calibration curve of an instrument (or the comparison between two instruments). So the x-axis is a known amount of something and the y-axis is the signal of the instrument. Both are a mean +/- standard deviation. Something like this openi.nlm.nih.gov/imgs/512/71/3867315/… (it's random google search). So I would expect a linear correlation. Slope and intercept are the key parameters, but I would probably want a measure of the correlation (if Pearson correlation coefficient is not computed by these type of models). – rs028 2 days ago