Let's say that I am fitting a linear regression model to predict random variable $Y$ based on random variable $X$, and I know as a fact that the only random variable that detemines the value of $Y$ is $X$. The value of $R^2$ describes how much of the variance in the samples from $Y$ is reduced by my model. The reason $R^2$ is not $1.0$ is two fold:

  1. The linear model does not fully capture how $X$ affects $Y$
  2. Measurement error

I want to remove the effect of measurement error from $R^2$, i.e. I like to know how much of the unexplained variance is caused because of the insufficiency of the model's complexity.

Let's say that for each $x$, I have multiple measurements of $y$ in the dataset. Can I use this to estimate the variance caused by measurement error and correct the value of $R^2$ accordingly? How does your answer generalize to deviance, for instance if I use GLM with Poisson distribution instead of the linear regression?

  • $\begingroup$ You are assuming that there is no irreducible variation in the universe (or at least in this system)? Is there measurement error in $X$, $Y$, or both? Do you have an external estimate of the measurement error? $\endgroup$ – gung - Reinstate Monica Feb 24 at 15:44
  • $\begingroup$ Yes, let's say that there is no variation in the system. The measurement error is only in $Y$, and there is no direct estimate of it. I am asking to estimate it based on different values of $y$ for a single $x$. $\endgroup$ – Feri Feb 24 at 18:04

If the measurement error is only in $Y$, it won't bias the fitted model, but only add residual noise (lowering power). With replicated measurements at the same values of $X$, you can conduct a lack of fit test. You don't want a function of sufficient complexity to spear every conditional mean—since the measurement error is (believed to be) real, that guarantees overfitting. Instead, the degree to which the conditional mean of $Y|X=x_i$ should bounce randomly around your model's $\hat{y}_{x_i}$ due to measurement error alone. That's what the case with a lack of fit test assesses. This can be done with either a linear model (OLS) or a non-normal GLM (e.g., a Poisson regression). Fit a larger model by adding a dummy for each unique value of $X$ that has multiple replicates, then perform a nested model test. If it's significant, the dummies are adding information that was missed by your primary model. Here's a simple example, coded in R:

d2 = sapply(with(corrosion, split(loss, Fe)), mean)
d2 = data.frame(Fe=f2n(names(d2)), loss=d2)
  plot(loss~Fe, corrosion)
  abline(lm(loss~Fe, corrosion), col="gray")
  text(1.75, 130, expression(R^2==0.9697))
  points(d2[,1], d2[,2], pch=3, col="lightcoral")

anova(lm(loss~Fe, corrosion), lm(loss~Fe+as.factor(Fe), corrosion))
# Analysis of Variance Table
# Model 1: loss ~ Fe
# Model 2: loss ~ Fe + as.factor(Fe)
#   Res.Df     RSS Df Sum of Sq      F   Pr(>F)   
# 1     11 102.850                                
# 2      6  11.782  5    91.069 9.2756 0.008623 **
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Scatterplot showing model tends to miss conditional means

  • $\begingroup$ Thank you. I lack some background and need to read more to fully understand how your solution provides me with the value I want. But one quick question. Did you assume in your solution that the measurement error is from a normal distribution? Does it also work if it is a Poisson distribution? (e.g if the model is simple linear regression but we know for a fact that we have measurement error in form of a Poisson distribution with mean equal to the estimation of the linear model) $\endgroup$ – Feri Feb 24 at 19:12
  • $\begingroup$ I'm not sure I follow that, @Feri. If you are fitting a Poisson GLM, that's fine, you can do the same thing as shown here. If you are fitting Poisson data w/ a linear regression, you are likely to have a poor model. $\endgroup$ – gung - Reinstate Monica Feb 24 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.