I am using the Deming function provided by Terry T. on this archived r-help thread. I am comparing two methods, so I have data that look like this:

y  x     stdy   stdx
1  1.2   0.23   0.67
2  1.8   0.05   0.89
4  7.5   1.13   0.44
... ...  ...   ...

I have done my Deming regression (also called "total least squares regression") and I get a slope and intercept. I would like to get a correlation coefficient so I've start calculating the $R^2$. I have manually entered the formula:

R2 <- function(coef,i,x,y,sdty){
    predy    <- (coef*x)+i
    stdyl    <- sum((y-predy)^2)   ### The calculated std like if it was a lm (SSres)
    Reelstdy <- sum(stdy)          ### the real stdy from the data  (SSres real)
    disty    <- sum((y-mean(y))^2) ### SS tot
    R2       <- 1-(stdyl/disty)    ### R2 formula
    R2avecstdyconnu <- 1-(Reelstdy/disty) ### R2 with the known stdy
    return(data.frame(R2, R2avecstdyconnu, stdy, Reelstdy))

This formula works and gives me output.

  • Which of the two $R^2$s makes more sense? (I personally think of both of them as kind of biased.)
  • Is there a way to get a correlation coefficient from a total least squared regression?


deming(x = Data$DS, y = Data$DM, xstd = Data$SES, ystd = Data$SEM,     dfbeta = T)

               Coef  se(coef)         z            p
Intercept 0.3874572 0.2249302 3.1004680 2.806415e-10
Slope     1.2546922 0.1140142 0.8450883 4.549709e-02

   Scale= 0.7906686 
  • 2
    $\begingroup$ Since a correlation coefficient can be computed without any reference to regression at all, could you clarify what you mean by "correlation coefficient"? I'm trying to figure out whether the Pearson $\rho$ will answer your question or if you are perhaps looking for some kind of "coefficient of determination" to serve as an analog for the least squares $R^2$. If it's the latter, what are you hoping this analog will tell you? $\endgroup$
    – whuber
    Commented Nov 30, 2015 at 21:02
  • $\begingroup$ I am trying to see how well y correlate with x. I've added my output to the question from the deming regression(total least squared regression) I want to be able to say that the two method give similar results in other words. $\endgroup$ Commented Nov 30, 2015 at 21:58

2 Answers 2


To elaborate on whuber's answer above - Pearson will give you what you want. It determines how well y correlates with x using an approach that is independent of regression model:


gx.rma from the rgr package will do total least squares and calculate Pearson for you (or you can continue with Deming and do it manually).


[1] 0.9922014

So, the basic answer to your question is, when doing total least squares, forget R-squared and just use Pearson. You can always square that if you want a result between 0 and 1. This will do everything you need.

Having said that, I will elaborate a little as I understand it feels like we should be able to calculate an R-squared equivalent.

First, let's try a normal sum of squares regression on the data using lm. Notice that it gives the same correlation coefficient as Pearson (after square rooting and only worrying about magnitude, obviously).

[1] 0.9922014

This is calculated from the lm model result using the traditional sum of squares approach


So, provided you use the model given by lm, (Pearson)-squared and R-squared are equivalent.

However, if you use the model from the total sum of squares regression, and try to use the latter equation, you will get a slightly different result. That's obvious because normal and total least squares use different minimisation functions so give models with slightly different gradients and intercepts. (Remember, the first equation will still give the same result as it is looking at the data only.)

This is where I get hung up though. If the two equations give the same result when using the lm model, then surely there must be an equivalent formulation for the latter equation, but when using the total least squares model, which also gives the same result?

I had a quick play around with different approaches using the appropriate minimisation function (as has the poster here: Coefficient of determination of a orthogonal regression), but cannot find a way to do it - if there is a way.

Perhaps we are both getting hung up on the fact that Pearson and R-squared give the same result when using normal least sqaures - and there simply isn't a way to do R-squared on total least squares, which will give the same result as Pearson. But I don't know enough about this to say why not.

  • $\begingroup$ Thanks for your answer but ... I still think the gx.rma is not appropriate for what I am trying to do that's why I've use deming regression. Because I don't get how the gx.rma actually account for the known standard error...But it might be me misunderstanding something here... I'll have to check it out when I get time $\endgroup$ Commented Dec 9, 2015 at 20:55
  • 1
    $\begingroup$ The point is that you can calculate a correlation coefficient between x and y (which you mention) without the need for any regression at all. I think you are hung up on the difference of a correlation between the data, and an assessment of the quality of the regression - they are not necessarily the same thing. Pearson^2 gives you the former and, in the case of normal-LS (with lm), is the same as R^2. For orthogonal-LS, they're not. So asking about correlation coeff. and R^2 (especially now you mention std error) are actually two different questions. You might want to rephrase the OP. $\endgroup$
    – Mooks
    Commented Dec 9, 2015 at 21:26
  • $\begingroup$ I might not have been clear enought....what I want to do is the Pearson. But what is bugging me is that there's no R^2 for the total least square regression. I use the total least square to adjust my coefficient (slope). But I'll use the Pearson for the correlation as you suggested . Thanks $\endgroup$ Commented Dec 9, 2015 at 21:38
  • $\begingroup$ I understand, it's something that is bugging me too! I think there should be an equivalent formula to give an R2 for total least squares too, but probably I am not understanding fully. Although, if you just want to adjust your slope, then you could just use the minimisation function that is the basis of total LS in Deming. Try mathworld.wolfram.com/… or arxiv.org/pdf/math/9805076.pdf. You could also consider principle component analysis with prcomp or princomp - it's basically the same and maybe they kick out something that helps. $\endgroup$
    – Mooks
    Commented Dec 9, 2015 at 21:49

Using the package "mcr"

and using function to generate your deming regression model

yourmodel<-mcreg(x, y, ...) # you need to be familiar with the various types of deming constant SD or CV%. these can give very different results. But that's different question.

and producing a plot using the function

MCResult.plot(your model)

This displays the Pearson's production moment correlation on the plot for the model, which tells you the strength and the direction of the linear relationship between your two x,y variables, but does not give the proportion of the variation that is explained.

Hope that helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.