# Performance of linear least squares regression subject to inequality (bounded interval) constraints on parameters

Consider the following model:

$${\bf y} = {\bf X}{\bf b} + {\bf e}$$

where ${\bf y}, {\bf n}\in {\cal R}^m$, ${\bf b}\in{\cal R}^n$, and ${\bf X}\in{\cal R}^{m \times n}$ where $m>n = {\rm rank}({\bf X})$. ${\bf e}$ is an error vector (with independent, identically distributed components of zero mean and variance $\sigma^2$).

Upon observation of ${\bf y}$, an estimate of ${\bf b} =[b_0, \cdots, b_{n-1}]^T$ (where $[\cdot]^T$ denotes transpose) is sought subject to bounding constraints on each parameter:

$$b_i \in [l_i,u_i]\quad i\in\{0,1,\cdots, n-1\}$$ There are numerical packages to solve such problems.

In the absence of these constraints, it is well-known that the least squares estimate of $\bf b$ is unbiased with covariance $\sigma^2 ({\bf X}^T {\bf X})^{-1}$.

Do simple results exist for the bias and covariance of the estimation error exist under the constrained parameter scenario?

In addition, assuming, say, normality of the error vector, are there simple expressions for the Cramer Rao Bound (CRB) of $\bf b$ under the parameter constraints?

I think I may have a solution at least for the CRB. As per http://davegiles.blogspot.com/2015/05/maximum-likelihood-estimation.html, I could non-linearly re-parameterize the unknowns such that the constraints are automatically satisfied:

$$b_i = f(\theta_i) = l_i + (u_i-l_i)e^{\theta_i}/(1+e^{\theta_i}).$$
The CRB of $\{\theta_i\}$ is easily calculated for normally distributed errors. Determination of the CRB of the transformation to a "new" set of variables $\{b_i\}$ is also easy. Any thoughts? Does this seem correct?

• Not a full answer, but with an appropriate shifting and rescaling of the data, this range constraint might be equivalent to ridge regression or the lasso on the subset of bounded parameters. Commented Dec 26, 2016 at 3:53
• How would you define the bias for $b_i$: in terms of its limitation to $[l_i,u_i]$, or in terms of possible values of $b_i$ over all real numbers? For example, if the "true" $b_i$ is much lower than $l_i$, the bias and variance for $b_i$ values limited to $[l_i,u_i]$ would probably be very small, in the sense of how the estimate would change over multiple replications of the same experiment. Nevertheless, the bias with respect to the "true" value, much lower than $l_i$, would be high.
– EdM
Commented Dec 27, 2016 at 20:28
• For this problem, assume that the true $b_i$ would never be lower than $l_i$. Rather, the true parameter is constrained to the interval $[l_i, u_i]$ and the estimator is given exact knowledge of the interval. That is: assume that there is no model mis-match in the sense that $b_i$ is guaranteed to reside in $[l_i, u_i]$.
– rhz
Commented Dec 27, 2016 at 20:51
• Bias is defined in the standard manner: bias = E[\hat{b}_i]-b_i where the expectation is over all realizations of the noise vector ${\bf e}$.
– rhz
Commented Dec 27, 2016 at 20:54
• As noted by @jwimberley your problem, at least the MAP estimate part, can be transformed to an equivalent LASSO problem. I do not believe there are any simple "standard error" formulas for LASSO. However Ch. 6 of Statistical Learning With Sparsity (2015) summarizes relevant approaches to quantifying parameter uncertainty for LASSO. Commented Dec 28, 2016 at 5:40

Now, regarding the covariance of the estimate: This is much trickier. It's quite likely that if the least-squares estimate of the parameter $\beta$ is outside the constraint range, the quadratic optimization value will be on the boundary. The covariance is likely to be ill-defined. There are two alternate methods you could consider: ridge regression and the lasso. There are popular packages for performing these types of regression. One interpretation of them is that they place an a priori probability density on the parameters. These probabilities are, as it happens, normal distributions and absolute-exponential distributions with means of 0, respectively, but you could redefine and adapt your problem so that either of these is suitable. Since the a priori pdfs are smooth, so are the posterior pdfs, and the covariances are well-defined.