References for weighted linear regression with linear constraints on the coefficients?

Question

I would like to estimate a multiple linear regression model $N$ observations (with $\beta$ of length $k$):

$$Y = X \beta + \epsilon$$

subject, however, to some linear constraints on the coefficients. I.e a constraint of the form

$$M \beta = 0$$

where (at least in my case) $M$ is of dimension $2 \times k$ but more generally can be expressed as

$$M \beta = c$$ where $M$ is of dimension $p \times k$ with $p < N$ and $c$ of dimension $p \times 1$.

I have not been able to find much about this online and would appreciate references to read up on this..

Thanks

Answer 1

The slides from the website below give a very detailed procedure on how to transform your equality constrained least squares into simply an unconstrained least squares problem. It uses QR decomposition to split the normal equations into a part that depends on the constraints and another part that does not. I hope this proves useful!

The slides you're looking for are on 19,20 & 21.

See: http://folk.uio.no/inf9540/CLS.pdf (warning: .pdf)

Answer 2

This could be solved using a standard convex optimization solver.

For the case of weighted regression, we usually try to minimize $\displaystyle\sum_{i,j} ((Y_{i,j} - X\hat{B}_{i,j})w_{ij})^2$ by choosing some $\hat{B}$. That is, we compute the matrix of error terms, weigh them by w, and then determine which $\hat{B}$ minimizes the Frobenius norm.

This objective function can be fed into a convex solver like CVX. Since your constraints are linear, they can also be used in the convex solver.