# Covariance of an estimate from optimization

Consider a standard linear regression model, $$\boldsymbol y = X \boldsymbol \beta + \boldsymbol \epsilon$$. $$\boldsymbol y$$ is a vector of $$m$$ responses, $$X$$ is a design matrix with $$m$$ rows and $$p$$ columns and noise term $$\epsilon \sim \mathcal{N}(0,\mathbb{I})$$ where $$\mathbb{I}$$ is an identity matrix. In this model the least squares estimate is $$\hat{\boldsymbol \beta} = (X^\top X)^{-1}X^\top\boldsymbol y$$. It's covariance is, $$\text{cov}(\hat{\boldsymbol \beta}) = (X^\top X)^{-1}$$.

It is also possible to get the estimate by minimizing the sum of squared residuals given by

$$\hat{\boldsymbol \beta} = \underset{\boldsymbol \beta} {\mathrm{argmin}}\ (y-X{\boldsymbol \beta})^\top(y-X{\boldsymbol \beta})$$

This problem can be solved analytically and one can still get the covariance of $$\hat{\boldsymbol \beta}$$.

However consider a different and slightly more difficult problem such as a constrained Lasso

\begin{aligned} \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta} {\mathrm{argmin}}&\ (y-X{\boldsymbol \beta})^\top(y-X{\boldsymbol \beta}) + \lambda || \beta ||_1 \\ \text{s.t.}&\ C\beta \leq \boldsymbol d \end{aligned} where $$C$$ is known matrix and $$\boldsymbol d$$ is a known vector. This optimization problem cannot be solved analytically. In such cases how does one compute the covariance of the estimate. Also, is there something better than covariance to examine the uncertainty in $$\hat \beta$$

• It depends on the constraints. When one or more constraints are "active" in the solution, the covariance matrix is a poor representation of the uncertainty in $\hat\beta,$ which is why one ought to hesitate to ask for or rely on a general answer. A special case of this question is the issue of obtaining p-values for the Lasso: see stats.stackexchange.com/… for some good discussions. – whuber May 23 at 14:29
• @ whuber I have made small changes to make the problem more specific – Rohit Arora May 23 at 16:51
• You can usually get stable estimates of covariance by using the Hessian matrix from the quasi-Newton procedure. I don't know about LASSO since the $\lambda$ penalty is a tuning parameter. It's not too hard to think about optimization problems where the Hessian is a reasonable approximation to the covariance matrix, like non-linear least squares. – AdamO May 25 at 14:03

You could bootstrap by solving the constrained Lasso problem for each bootstrap sample. That is probably the most meaningful thing to do.

Alternatively, you could look at Covariance projected into the Jacobian of the null space of the active constraints, i.e., those constraints satisfied with equality at the optimum. If $$Z$$ is a basis for the Jacobian of the nullspace of active constraints, and $$K$$ is the unconstrained Covariance, then the projected Covariance = $$Z^TKZ$$. The unconstrained covariance $$K$$ is obtained by the inverse Hessian of the objective function calibrated per the Residual Sum of Squares.

Projected covariance amounts to looking at uncertainty when (i.e., subject to, i.e., given that) the constraints which are active at the optimum remain active and not accounting for the possibility that other constraints become active - so it's a very limited type of uncertainty, and is really only locally valid. So I think bootstrap is better if computationally doable.