6
$\begingroup$

I would like to estimate an ordinary least squares regression of the form $$ y = X\beta + \varepsilon, \ $$ except that, instead of minimizing the sum of squared residuals, $$ SSR(b)=(y-Xb)'(y-Xb) $$ I want to minimize $$ (y-Xb)'(y-Xb)+\lambda(b-\tilde{\beta})'(b-\tilde{\beta}) $$ where $\lambda$ is some constant. Otherwise, notation above is as on wikipedia. All the standard assumptions are satisfied.

Is there some way I can modify the regression to perform the joint minimization?

$\endgroup$
7
$\begingroup$

Differentiating the objective function with respect to $b$ and equating to $0$ shows that the solution to the modified equation is obtained by solving

$$(X'X + \lambda)b = X'y + \lambda\tilde{\beta}.$$

If your software won't do that directly, you can get the same results with this trick:

  • Include a column of 1's in the dataset to explicitly model the constant. Do the fitting without a constant term.

  • For $p$ independent variables (including the constant), include $p$ additional fake data. For fake case $i$, $i=1,\ldots,p$, set $X_i = \sqrt{\lambda}$, $y = \sqrt{\lambda}\tilde{\beta}$, and all other $X_j=0$. (Of course we require $\lambda \ge 0$.)

Although you can obtain the solution $\hat{b}$ this way, I doubt any of the statistics coming out of this fit will be meaningful.

$\endgroup$
2
$\begingroup$

This looks a lot like ridge regression, the lm.ridge function in the MASS package for R does ridge regression and the ols function in the MASS package also does penalized regression. If neither of those does exactly what you want they could be used as a starting point. You could also look at the lasso and lars algorithms (there are packages for these for R as well) which uses an L1 penalty term instead of L2.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.