I read that, for the group lasso, to solve the zero subgradient equations, one approach involves keeping all block vectors fixed, denoted as $\{\hat\theta_k, k \ne j\}$, and then solving for $ \hat \theta_j$. This procedure corresponds to employing block coordinate descent on the group lasso objective function. Given the convex nature of the problem and the block-separable penalty, convergence to an optimal solution is guaranteed (Tseng 1993). With all $\{\hat\theta_k, k \ne j\}$ held constant, the equation becomes:

\begin{equation} -\mathrm{\it{Z}}_{j}^{T}(r_{j}-\mathrm{\it{Z}}_{j}\widehat{\theta}_{j})+\lambda\widehat{s}_{j}=0, \end{equation}

From the conditions satisfied by the subgradient $\hat s_j$, that is the subgradient of the euclidean norm, we must have $\hat \theta = 0$ if $||Z_j^Tr_j||_2 < \lambda$, and otherwise the minimizer $\hat \theta_j$ must satisfy:

\begin{equation} \widehat{\theta}_{j}=\left(\mathbf{Z}_{j}^{T}\mathbf{Z}_{j}+{\frac{\lambda}{||\widehat{\theta}_{j}||_{2}}} I \right)^{-1}\mathbf{Z}_{j}^{T}r_{j}. \end{equation}

It's important to note that the above equation lacks a closed-form solution for $\hat \theta_j$ unless $ Z_j$ is orthonormal. IN THis special case, we have the simple update:

\begin{equation} \widehat{\theta}_{j}=({1-\frac{\lambda}{||Z_j^Tr_j||_2})_{+} Z_j^Tr_j} \end{equation}

where $(t)_+ := max\{0,t\}$ is the positive part of the function. I do not understand where does this last formula come from. What happens when we have orthonormality? How do we arrive to define equation 3?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.