# References for weighted linear regression with linear constraints on the coefficients?

I would like to estimate a multiple linear regression model $N$ observations (with $\beta$ of length $k$):

$$Y = X \beta + \epsilon$$

subject, however, to some linear constraints on the coefficients. I.e a constraint of the form

$$M \beta = 0$$

where (at least in my case) $M$ is of dimension $2 \times k$ but more generally can be expressed as

$$M \beta = c$$ where $M$ is of dimension $p \times k$ with $p < N$ and $c$ of dimension $p \times 1$.

I have not been able to find much about this online and would appreciate references to read up on this..

Thanks

-
It's not quite clear what your problem is here. What are these "other weights z"? Also, it sounds like your issue with the constraints on beta is actually the problem rather than anything to do with the weighting. So perhaps you should be a bit more specific about what those constraints are. –  Peter Ellis Jan 19 '12 at 18:19
I've edited my question...I basically want to know where I can read about/ learn about regression with these types of constraints. –  Palace Chan Jan 19 '12 at 18:56
I'm not the swiftest with mathematical notation...what is p? What is the significance of p being multiplied by 1? And is this a mixture design? –  rolando2 Jan 19 '12 at 19:46
@rolando2: Not a mixture model..and p isnt multiplied by 1..it's just px1 in dimensions...no particular significance im just saying some of the coefficients to be estimated must satisfy linear constraints..in the above notation $p$ just happens to be the number of such linear constriants. –  Palace Chan Jan 19 '12 at 22:25
Pretty much any linear regression theory textbook will provide the derivation and (closed-form) solution for finding the least-squares estimates of the coefficients under linear equality constraints. See, for example, G. A. F. Seber and A. J. Lee, Linear Regression Analysis, 2nd. ed., Sec. 3.8, pp 59-62. –  cardinal Jan 20 '12 at 2:27
add comment

## 2 Answers

The slides from the website below give a very detailed procedure on how to transform your equality constrained least squares into simply an unconstrained least squares problem. It uses QR decomposition to split the normal equations into a part that depends on the constraints and another part that does not. I hope this proves useful!

The slides you're looking for are on 19,20 & 21.

See: http://folk.uio.no/inf9540/CLS.pdf (warning: .pdf)

-
Thank you for the reference. Welcome to our site! –  whuber Jan 20 '12 at 14:34
add comment

This could be solved using a standard convex optimization solver.

For the case of weighted regression, we usually try to minimize $\displaystyle\sum_{i,j} ((Y_{i,j} - X\hat{B}_{i,j})w_{ij})^2$ by choosing some $\hat{B}$. That is, we compute the matrix of error terms, weigh them by w, and then determine which $\hat{B}$ minimizes the Frobenius norm.

This objective function can be fed into a convex solver like CVX. Since your constraints are linear, they can also be used in the convex solver.

-
The current incarnation of this question is simply asking for solving a least-squares problem with linear equality constraints. There is a closed form solution, so I don't see the need for using a general convex solver here. (Maybe you were responding to some previous version of the question?) –  cardinal Jan 20 '12 at 2:21
Ah, my apologies. Can you post a link to the closed form solution? –  JCWong Jan 20 '12 at 2:28
No apology necessary. This question has undergone at least a couple edits. A previous version asked something for which this particular answer might be more relevant (though I believe that question didn't lend itself directly to a convex-programming solution; it looked biconvex to me at first blush). I've left a comment to the OP with one reference, but it is but one of many. Cheers. :) –  cardinal Jan 20 '12 at 2:29
add comment