# Covariance of an estimate from optimization

Consider a standard linear regression model, $$\boldsymbol y = X \boldsymbol \beta + \boldsymbol \epsilon$$. $$\boldsymbol y$$ is a vector of $$m$$ responses, $$X$$ is a design matrix with $$m$$ rows and $$p$$ columns and noise term $$\epsilon \sim \mathcal{N}(0,\mathbb{I})$$ where $$\mathbb{I}$$ is an identity matrix. In this model the least squares estimate is $$\hat{\boldsymbol \beta} = (X^\top X)^{-1}X^\top\boldsymbol y$$. It's covariance is, $$\text{cov}(\hat{\boldsymbol \beta}) = (X^\top X)^{-1}$$.

It is also possible to get the estimate by minimizing the sum of squared residuals given by

$$\hat{\boldsymbol \beta} = \underset{\boldsymbol \beta} {\mathrm{argmin}}\ (y-X{\boldsymbol \beta})^\top(y-X{\boldsymbol \beta})$$

This problem can be solved analytically and one can still get the covariance of $$\hat{\boldsymbol \beta}$$.

However consider a different and slightly more difficult problem such as a constrained Lasso

\begin{aligned} \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta} {\mathrm{argmin}}&\ (y-X{\boldsymbol \beta})^\top(y-X{\boldsymbol \beta}) + \lambda || \beta ||_1 \\ \text{s.t.}&\ C\beta \leq \boldsymbol d \end{aligned} where $$C$$ is known matrix and $$\boldsymbol d$$ is a known vector. This optimization problem cannot be solved analytically. In such cases how does one compute the covariance of the estimate. Also, is there something better than covariance to examine the uncertainty in $$\hat \beta$$

• It depends on the constraints. When one or more constraints are "active" in the solution, the covariance matrix is a poor representation of the uncertainty in $\hat\beta,$ which is why one ought to hesitate to ask for or rely on a general answer. A special case of this question is the issue of obtaining p-values for the Lasso: see stats.stackexchange.com/… for some good discussions.
– whuber
May 23, 2019 at 14:29
• @ whuber I have made small changes to make the problem more specific May 23, 2019 at 16:51
• You can usually get stable estimates of covariance by using the Hessian matrix from the quasi-Newton procedure. I don't know about LASSO since the $\lambda$ penalty is a tuning parameter. It's not too hard to think about optimization problems where the Hessian is a reasonable approximation to the covariance matrix, like non-linear least squares. May 25, 2019 at 14:03

Alternatively, you could look at Covariance projected into the Jacobian of the null space of the active constraints, i.e., those constraints satisfied with equality at the optimum. If $$Z$$ is a basis for the Jacobian of the nullspace of active constraints, and $$K$$ is the unconstrained Covariance, then the projected Covariance = $$Z^TKZ$$. The unconstrained covariance $$K$$ is obtained by the inverse Hessian of the objective function calibrated per the Residual Sum of Squares.