Covariance of an estimate from optimization

Question

Consider a standard linear regression model, $\boldsymbol y = X \boldsymbol \beta + \boldsymbol \epsilon$. $\boldsymbol y$ is a vector of $m$ responses, $X$ is a design matrix with $m$ rows and $p$ columns and noise term $\epsilon \sim \mathcal{N}(0,\mathbb{I})$ where $\mathbb{I}$ is an identity matrix. In this model the least squares estimate is $\hat{\boldsymbol \beta} = (X^\top X)^{-1}X^\top\boldsymbol y$. It's covariance is, $\text{cov}(\hat{\boldsymbol \beta}) = (X^\top X)^{-1}$.

It is also possible to get the estimate by minimizing the sum of squared residuals given by

$$ \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta} {\mathrm{argmin}}\ (y-X{\boldsymbol \beta})^\top(y-X{\boldsymbol \beta}) $$

This problem can be solved analytically and one can still get the covariance of $\hat{\boldsymbol \beta}$.

However consider a different and slightly more difficult problem such as a constrained Lasso

$$ \begin{aligned} \hat{\boldsymbol \beta} = \underset{\boldsymbol \beta} {\mathrm{argmin}}&\ (y-X{\boldsymbol \beta})^\top(y-X{\boldsymbol \beta}) + \lambda || \beta ||_1 \\ \text{s.t.}&\ C\beta \leq \boldsymbol d \end{aligned} $$ where $C$ is known matrix and $\boldsymbol d$ is a known vector. This optimization problem cannot be solved analytically. In such cases how does one compute the covariance of the estimate. Also, is there something better than covariance to examine the uncertainty in $\hat \beta$

Answer 1

You could bootstrap by solving the constrained Lasso problem for each bootstrap sample. That is probably the most meaningful thing to do.

Alternatively, you could look at Covariance projected into the Jacobian of the null space of the active constraints, i.e., those constraints satisfied with equality at the optimum. If $Z$ is a basis for the Jacobian of the nullspace of active constraints, and $K$ is the unconstrained Covariance, then the projected Covariance = $Z^TKZ$. The unconstrained covariance $K$ is obtained by the inverse Hessian of the objective function calibrated per the Residual Sum of Squares.

Projected covariance amounts to looking at uncertainty when (i.e., subject to, i.e., given that) the constraints which are active at the optimum remain active and not accounting for the possibility that other constraints become active - so it's a very limited type of uncertainty, and is really only locally valid. So I think bootstrap is better if computationally viable.