1
$\begingroup$

I read that, for the group lasso, to solve the zero subgradient equations, one approach involves keeping all block vectors fixed, denoted as $\{\hat\theta_k, k \ne j\}$, and then solving for $ \hat \theta_j$. This procedure corresponds to employing block coordinate descent on the group lasso objective function. Given the convex nature of the problem and the block-separable penalty, convergence to an optimal solution is guaranteed (Tseng 1993). With all $\{\hat\theta_k, k \ne j\}$ held constant, the equation becomes:

\begin{equation} -\mathrm{\it{Z}}_{j}^{T}(r_{j}-\mathrm{\it{Z}}_{j}\widehat{\theta}_{j})+\lambda\widehat{s}_{j}=0, \end{equation}

From the conditions satisfied by the subgradient $\hat s_j$, that is the subgradient of the euclidean norm, we must have $\hat \theta = 0$ if $||Z_j^Tr_j||_2 < \lambda$, and otherwise the minimizer $\hat \theta_j$ must satisfy:

\begin{equation} \widehat{\theta}_{j}=\left(\mathbf{Z}_{j}^{T}\mathbf{Z}_{j}+{\frac{\lambda}{||\widehat{\theta}_{j}||_{2}}} I \right)^{-1}\mathbf{Z}_{j}^{T}r_{j}. \end{equation}

It's important to note that the above equation lacks a closed-form solution for $\hat \theta_j$ unless $ Z_j$ is orthonormal. IN THis special case, we have the simple update:

\begin{equation} \widehat{\theta}_{j}=({1-\frac{\lambda}{||Z_j^Tr_j||_2})_{+} Z_j^Tr_j} \end{equation}

where $(t)_+ := max\{0,t\}$ is the positive part of the function. I do not understand where does this last formula come from. What happens when we have orthonormality? How do we arrive to define equation 3?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.