Can we sacrifice the estimation accuracy of some regression coefficients in order to gain accuracy in estimating the coefficients of primary interest?

Question

Let's say I have some regression model, say $\mathbb E[y]=\beta_0+\beta_1x_1+\beta_2x_2$. However, I am fitting this regression because I want to estimate $\beta_1$ while "controlling for" the effect of $x_2$, as is common.

A typical approach might be to fit the regression with a method like least squares or maximum likelihood estimation in order to estimate the entire parameter vector $\beta = (\beta_0,\beta_1,\beta_2)$. However, if I do not care about making an accurate estimate of $\beta_2$ or even $\beta_0$, is there a way to trade off some accuracy in the estimation of those coefficients in order to improve accuracy in estimating the $\beta_1$ of interest?

Answer 1

Below is an example of ridge regression for a linear model with two negatively correlated regressors. We can make the decision to penalize both coefficients or only one the first coefficient.

Based on the simulations it seems that the first coefficient has a lower least square error if we decide to penalize only the first coefficient and not the second coefficient as well. This is at the cost of the least square error of the second coefficient.

Output of the R-code below

Average coefficient estimate (the true coefficient is 1 and the higher 0.7438648 value mean less shrinking)

    One regularised     Both regularised
    beta_1    beta_2    beta_1    beta_2
[1] 0.6361610 0.7438648 0.6021872 0.6015797

Least squares error

For the first coefficient: If we only regularise the first coefficient then the 0.419 value is lower than the 0.464 mean squared error value for the situation where we regularise both.

For the second coefficient: the story is the other way around. While the mean squared error for the first coefficient is improved it is decreased for the second coefficient.

    One regularised     Both regularised
    beta_1    beta_2    beta_1    beta_2
[1] 0.4191937 0.5506929 0.4639312 0.4639455

This situation is only the case for negatively correlated regressors. So this effect will depend on the situation. It is not unimaginable that for each type of correlation there is a type of balance between penalties that optimizes the error for one of the coeffients at the cost of the error in the other.

Code:

library(MASS)
n = 5
rho = -0.7

fit = function(par, xt,yt,xv,yv, both=1, r = 0) {
    ### perform training with ridge regression 
    M = t(xt) %*% xt + par^2*matrix(c(1,0,0,both),2) 
    beta = solve(M) %*% t(xt) %*% yt

    ### perform validation for selection of ideal 'par' value
    fit_y = xv %*% beta
    error = sum((fit_y-yv)^2)

    if (r==0) {
       return(error)  # return optimal penalty 
    } else {
       return(beta)   # return beta at optimal penalty
    }
}


beta = function() {

  ### training data
  xt = mvrnorm(n, c(0,0), matrix(c(1,rho,rho,1),2))
  yt = rowSums(xt) + rnorm(n)

  ### validation data
  xv = mvrnorm(n, c(0,0), matrix(c(1,rho,rho,1),2))
  yv = rowSums(xv) + rnorm(n)

  ### optimize ridge regression
  ### penalizing only on coefficient 
  opt = optim(0,fit,xt=xt,yt=yt,xv=xv,yv=yv, both = 0)
  ### optimize ridge regression
  ### penalizing both coefficients 
  opt2 = optim(0,fit,xt=xt,yt=yt,xv=xv,yv=yv, both = 1)

  b1 = fit(opt$par,xt=xt,yt=yt,xv=xv,yv=yv, both = 0, r=1)
  b2 = fit(opt2$par,xt=xt,yt=yt,xv=xv,yv=yv, both = 1, r=1)
  return(c(b1[1],b1[2],b2[1],b2[2]))
}

 
set.seed(1)
b = replicate(10^4, beta())


rowMeans((b-1)^2)
rowMeans(b)