3
$\begingroup$

I'm aware there's a great number of questions which deal with the mathematical difference between the two, but I'm still confused as to best practice.

Basically I'm looking at a situation where we have lots of bad data, and some good data (which we can assume accurately measures the variable we're concerned with). The current approach is to regress the good data onto the bad data for what is basically a training set, i.e. $$good = m*bad+c+\epsilon$$

This is what I mean by reverse regression. We then use the ample bad data to predict what the good data would have been if it were collected elsewhere (think of this as the testing stage).

When I first saw this, I was convinced it was wrong. The bad data has the measurement errors, and the 'correct' approach is to fit a model regressing the bad data onto the good data, then invert the model to predict in the opposite direction. This is what I mean by inverse regression.

However I've seen since been playing with some toy models and I'm starting to see that the 'wrong' method produces much better predictions.

n <- 1000
x <- runif(n,-10,10)
m <- 0.7
c <- 4
e <- rnorm(n,0,3)
y <- m*x+c+e
df <- data.frame(x=x,y=y)
traindf <- df[1:(4*n/5),]
testdf <- df[(4*n/5+1):n,]
mod1 <- lm(y ~ x, data=traindf)
mod2 <- lm(x ~ y, data=traindf)
preds1 <- (testdf$y-coef(mod1)[1])/coef(mod1)[2]
preds2 <- predict(mod2, newdata=testdf)
mean((preds1-testdf$x)^2)
[1] 18.62054
mean((preds2-testdf$x)^2)
[1] 12.50204

Now I get why this is the case, model 2 is designed to reduce mean square error when predicting x from y, it's hardly surprising it does a better job at doing it. And I also understand the geometrical interpretations, one model is minimising vertical distances between the points and the line-of-best fit, whereas the other is minimising the horizontal differences etc. I get the maths.

What I don't understand is, why should I care that the second model is misspecified if it's giving me better predictions regardless?

Are there any advantages to using the 'correct' model? Are there other situations where inverse regression actually outperforms? Maybe there are other loss functions it does well with, or it's more robust to violations of assumptions, or more wrong but less biased. But so far it looks like reverse regression produces better point estimates and I'm going to bootstrap my confidence intervals in any case. It's also less of a hassle than manually inverting your model or playing around with the library investr.

What am I missing? This isn't an academic question as the data produced matters somewhat.

$\endgroup$
0
$\begingroup$

The problem here is that the variables are not standardized (scaled).

I scaled x and y values and carried out the regression and reverse regression as below:

n <- 1000
set.seed(23658)
x <- runif(n,-10,10)
m <- 0.7
c <- 4
e <- rnorm(n,0,3)
y <- m*x+c+e
df <- data.frame(x=x,y=y)
df<- scale(df)
df<- as.data.frame(scale(df))

mod1 <- lm(y ~ 0+x, data=df)
mod2 <- lm(x ~ 0+y, data=df)
mean((mod1$residuals)^2)
[1] 0.3673545
mean((mod2$residuals)^2)
[1] 0.3673545

I didn't split the data into train and test.

Note: In your code, the first variable that is detected should be y instead of x, as y is the dependent variable.

mean((preds1-testdf$y)^2)
mean((preds2-testdf$x)^2)

Hope this answers your query.

$\endgroup$
  • 1
    $\begingroup$ "In your code, the first variable that is detected should be y instead of x, as y is the dependent variable." This has been done on purpose by JackEm. The idea is to use both regression lines to predict $x$ as function of $y$. You did not do this, and instead compare the residuals of the two regression lines. $\endgroup$ – Sextus Empiricus Nov 7 '20 at 20:01
0
$\begingroup$

The regression lines do not relate to the true causal relationship (like your $good = m*bad+c+\epsilon$), but instead they relate to the conditional distribution of the one variable based on the other.

This will be a different line for $x$ as function of $y$ in comparison to $y$ as function of $x$. The image below illustrates this very well (the image is from the question: Effect of switching response and explanatory variable in simple linear regression)

example

The lines for $E(Y|X)$ and $E(X|Y)$ are not the same. So if you do the regression in the wrong direction and then invert the relationship, then you will get a biased result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.