# How to perform Least Squares with constraints on a subset of the model coefficients?

For solving an unconstrained LS regression $$\hat{y}=w_1.x_1+w_2.x_2+w_3.x_3+w_4.x_4 + \epsilon$$ I use the following normal equation: $$W^*=(X^{\top}X)^{-1}X^{\top}Y$$

If I want to introduce a constraint on all of the parameters, e.g. the ridge regression $w_i^2<c$, for $\forall i$, I can formulate the Lagrangian:

$$\mathcal{L(W,\lambda)} = (Y - XW)^2 - \lambda (W)^2.$$

and also obtain a matrix form solution, given by: $$W^*=(X^{\top}X-\lambda I)^{-1}X^{\top}Y$$

The question is: how may I formulate the above equation if I only want to impose constraints on some of the coefficients, e.g. $w_1=w_2$ with $w=[w_1, w_2, ..., w_d]$?

I can get to the Lagrangian, which would be: $$\mathcal{L(W,\lambda)} = (Y - XW)^2 - \lambda (w_1-w_2)$$ but I can't get to the matrix solution for $W^*$.

I'm searching for a manual solution (i.e. with no python or R code). Thanks in advance and sorry for the not-rigorous notation.

• With $w_1=w_2$, just look at the linear predictor $w_0+w_1 x_1 + w_2 x_2 + \dotsm = w_0 + w_1 (x_1+x_2) + \dotsm$! So jus remove one of them, and add the corresponding predictors. Here is a similar but more complicated example: stats.stackexchange.com/questions/248779/… Jun 27, 2018 at 9:42
• What could I do to impose a constraint that w1 should be "around" 1000 and w3 "around" 50? Jan 30, 2019 at 5:59
• Maybe a Bayes solution, with a prior distribution with mean of $W1$ 1000 and of $w3$ 50? and some prior variance expressing how certain you are about those restrictions. Or, not going that route, using regularization but with offsets in addition, that is , $w1$ is represented by w1+offset(Id(50*w1)) in R. Jan 30, 2019 at 7:29
• Given your comment "What could I do to impose a constraint that w1 should be "around" 1000 and w3 "around" 50" - would you like to shrink your coefficients towards particular prior centers? If so look at the answer here stats.stackexchange.com/questions/76925/… Apr 12 at 13:15
• If you are looking for least squares with box constraints on individual coefficients you can look at my answer here stats.stackexchange.com/questions/136563/… Apr 12 at 13:16

Ridge is the exception, not the norm, in this case. Applying a quadratic term in the Lagrangian still allows for a closed solution of the regression, and is one of the reasons why Ridge is interesting. Using a different function to regularize will most likely not produce a closed form solution. As an example, L1-regularization, the LASSO, has no closed form solution for $W^*$.