Lets say i want to identify the effect of some categorical variables x1 x2 x3 on a numeric y. I got data from 3 waves on the same individuals, but there are missing values on some variables because of two things:

  • x2 is measured only in wave 2 (for example some kind of personality test)
  • x3 is missing for individuals with x1 == 0 (for example x1 is employment status (employed yes or no) and x3 is occupational status(isco or something like that)

Doing a linear regression on y, listwise deletion will eliminate all individuals in wave 1 and 3 as well as everyone with x1 == 0.

What would be the best approach to get reliable estimates for all variables of interest?

Would it be correct to set the missing data to some arbitrary value, e.g. x2 == 88 for wave 1 and 3 (as "not measured" category) and x3 == 99 for people with x1 == 0 (as "not applicable" category) and include it in the regression? I would think the autocorrelation biases my estimates, as the x2=88 is essentially a year effect and x1 and x3 are now correlated highly as well.

Or should i estimate different models for my variables, e.g. one for x3 on y for wave 1 to 3 and one for x2 on y for wave 2?

Some R-Code for plausibility:


x1 <- round(runif(1000,0,1))
x2 <- round(runif(1000,1,5))
x3 <- round(runif(1000,1,5))
y <- round(x1*5+x2*5+x3*5+runif(1000,0,10))
w <- round(runif(1000,1,3))
dat <- as.data.frame(cbind(x1,x2,x3,y,w))
dat$x2[dat$w != 2] <- NA
dat$x3[dat$x1 == 1] <- NA
m1 <- lm(y~as.factor(x1)+as.factor(w), data=dat)
m2 <- lm(y~as.factor(x3)+as.factor(w), data=dat)
m3 <- lm(y~as.factor(x1)+as.factor(x2), data=dat)
m4 <- lm(y~as.factor(x3)+as.factor(x2), data=dat)

dat$x2[is.na(dat$x2)] <- 88
dat$x3[is.na(dat$x3)] <- 99
mf <- lm(y~as.factor(x1)+as.factor(x2)+as.factor(x3)+as.factor(w), data=dat)

The coefficient of x1 is rather different in mf vs. m1, though the x'es are uncorrelated without its relevance for y and in both regressions all observations are used.

This is directly related to 1 but in my opinion not solved, as there was just one method suggested and no real description of the mechanism. I mean, this seems like some real fundamental "problem" and something i seem to have missed in introductory statistics, as there should be a relatively trivial and intuitive answer.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.