# Can we sacrifice the estimation accuracy of some regression coefficients in order to gain accuracy in estimating the coefficients of primary interest?

Let's say I have some regression model, say $$\mathbb E[y]=\beta_0+\beta_1x_1+\beta_2x_2$$. However, I am fitting this regression because I want to estimate $$\beta_1$$ while "controlling for" the effect of $$x_2$$, as is common.

A typical approach might be to fit the regression with a method like least squares or maximum likelihood estimation in order to estimate the entire parameter vector $$\beta = (\beta_0,\beta_1,\beta_2)$$. However, if I do not care about making an accurate estimate of $$\beta_2$$ or even $$\beta_0$$, is there a way to trade off some accuracy in the estimation of those coefficients in order to improve accuracy in estimating the $$\beta_1$$ of interest?

• Not if you use a linear estimation technique - Gauss-Markov shows this. So, if such a method exists, it has to be nonlinear. Commented Feb 28, 2023 at 19:14
• @jbowman What if that linear estimator is biased for the coefficients of non-interest? Then Gauss-Markov does not apply, as we are not considering an unbiased estimator.
– Dave
Commented Feb 28, 2023 at 19:25
• Good catch, my apologies. Commented Feb 28, 2023 at 19:29
• Perhaps you may simply find more (causal) paths linking your interested variable $x_1$ and $y$ via some other mediated variable(s) thus you can get more reliable and quality correlation coefficient data to fix your $\beta_1$ more accurately hopefully. Commented Feb 28, 2023 at 22:18
• I guess that a linear estimator should be unbiased unless the model is misspecified like lacking regressors or the $x$ variable is correlated with the error. But one deviation could be to use a non-linear estimator like some regularisation plus cross validation. A tricky part is that the cross validation typically validates the error in predictions of $y$ and not error in $\beta$. I imagine that regularisation of only the $\beta_1$ coefficient might be a trick? Commented Mar 12 at 12:33

Below is an example of ridge regression for a linear model with two negatively correlated regressors. We can make the decision to penalize both coefficients or only one the first coefficient.

Based on the simulations it seems that the first coefficient has a lower least square error if we decide to penalize only the first coefficient and not the second coefficient as well. This is at the cost of the least square error of the second coefficient.

Output of the R-code below

Average coefficient estimate (the true coefficient is 1 and the higher 0.7438648 value mean less shrinking)

    One regularised     Both regularised
beta_1    beta_2    beta_1    beta_2
[1] 0.6361610 0.7438648 0.6021872 0.6015797


Least squares error

For the first coefficient: If we only regularise the first coefficient then the 0.419 value is lower than the 0.464 mean squared error value for the situation where we regularise both.

For the second coefficient: the story is the other way around. While the mean squared error for the first coefficient is improved it is decreased for the second coefficient.

    One regularised     Both regularised
beta_1    beta_2    beta_1    beta_2
[1] 0.4191937 0.5506929 0.4639312 0.4639455


This situation is only the case for negatively correlated regressors. So this effect will depend on the situation. It is not unimaginable that for each type of correlation there is a type of balance between penalties that optimizes the error for one of the coeffients at the cost of the error in the other.

Code:

library(MASS)
n = 5
rho = -0.7

fit = function(par, xt,yt,xv,yv, both=1, r = 0) {
### perform training with ridge regression
M = t(xt) %*% xt + par^2*matrix(c(1,0,0,both),2)
beta = solve(M) %*% t(xt) %*% yt

### perform validation for selection of ideal 'par' value
fit_y = xv %*% beta
error = sum((fit_y-yv)^2)

if (r==0) {
return(error)  # return optimal penalty
} else {
return(beta)   # return beta at optimal penalty
}
}

beta = function() {

### training data
xt = mvrnorm(n, c(0,0), matrix(c(1,rho,rho,1),2))
yt = rowSums(xt) + rnorm(n)

### validation data
xv = mvrnorm(n, c(0,0), matrix(c(1,rho,rho,1),2))
yv = rowSums(xv) + rnorm(n)

### optimize ridge regression
### penalizing only on coefficient
opt = optim(0,fit,xt=xt,yt=yt,xv=xv,yv=yv, both = 0)
### optimize ridge regression
### penalizing both coefficients
opt2 = optim(0,fit,xt=xt,yt=yt,xv=xv,yv=yv, both = 1)

b1 = fit(opt$$par,xt=xt,yt=yt,xv=xv,yv=yv, both = 0, r=1) b2 = fit(opt2$$par,xt=xt,yt=yt,xv=xv,yv=yv, both = 1, r=1)
return(c(b1[1],b1[2],b2[1],b2[2]))
}

set.seed(1)
b = replicate(10^4, beta())

rowMeans((b-1)^2)
rowMeans(b)