# Maximizing (and derivating) log-likelihood of penalized logistic regression

I'm trying to solve Exercise 18.3 of "Elements of Statistical learning" by Hastie et al. and I'd be really grateful for any hints.

Show that the fitted coefficients for the regularized multiclass logistic Regression Problem $$\max_{\{\beta_{0k},\beta_{k}\}_{1}^{K}}\left[\sum\limits_{i=1}^{N}logPr(g_{i}| x_i)-\frac{\lambda}{2}\sum\limits_{k=1}^{K}\|\beta_{k}\|_{2}^{2}\right]$$ satisfy $$\sum\limits_{k=1}^K\hat{\beta}_{kj}=0, j = 1,...,p.$$

$p$ is the number of Features, $K$ is the number of classes.

EDIT: I actually think that $\max\limits_{\{\beta_{0k},\beta_{k}\}_{1}^{K}}\left[\sum\limits_{i=1}^{N}logPr(g_{i}| x_i)-\frac{\lambda}{2}\sum\limits_{k=1}^{K}\|\beta_{k}\|_{2}^{2}\right]$ is a Lagrange optimization Problem, so derivating with respect to $x$ and $\lambda$ and then Setting to Zero should do what I Need. But turns out that I'm even getting confused by derivating! $$Pr(g_{i}| x_i) = \dfrac{exp(\beta_{g_i0}+x_i^T\beta_{g_i})}{\sum\limits_{l=1}^Kexp(\beta_{l0}+x_i^T\beta_l)}$$ and I'm not quite sure if I should compute $$\dfrac{\partial f}{\partial x_i}$$ with $i=1,...,N$ or $$\dfrac{\partial f}{\partial x_{ij}}$$ with $i=1,...,N,j=1,...,p$, $f$ being the function inside the square brackets. Well I tried both and I guess it went wrong both times as I couldn't achieve what I'm trying to proove. I didn't forget $\dfrac{\partial f}{\partial \lambda}.$

Hint #1 - assume you have a set of betas which does not sum to zero, but instead you have $\sum_{k=1}^K\beta_{kj}=c$.
Hint #2 - what happens to the first term $\log [Pr(g_i|x_i)]$ if you add the same constant to each intercept term $\beta_{0k }=\beta_{0k}+d$ ?
• Regarding Hint #2: Nothing happens to the first term, $Pr(G=k|X=x)=\dfrac{exp(\beta_{k0} + x^T\beta_k)}{\sum\limits_{l=1}^Kexp(\beta_{l0} + x^T\beta_l)}$ in this case, so the additional d cuts itself out. – machinelearner Dec 15 '17 at 11:36
• Do you want me to regard $\sum\limits_{k=1}^K\beta_{kj}$ as a constant $d$ that's added to $\beta_{0k}$? – machinelearner Dec 15 '17 at 19:09
• Does the second term vary if you add a constant $e$ to each term. Is there an optimal choice of what to add? – probabilityislogic Dec 15 '17 at 22:47
• How do you get $e=0$ as the best choice? This is least squares 101... best choice for $e$, take derivative and set to zero. Solve for $e$... – probabilityislogic Dec 16 '17 at 11:18