1
$\begingroup$

I'm trying to solve Exercise 18.3 of "Elements of Statistical learning" by Hastie et al. and I'd be really grateful for any hints.

Show that the fitted coefficients for the regularized multiclass logistic Regression Problem $$\max_{\{\beta_{0k},\beta_{k}\}_{1}^{K}}\left[\sum\limits_{i=1}^{N}logPr(g_{i}| x_i)-\frac{\lambda}{2}\sum\limits_{k=1}^{K}\|\beta_{k}\|_{2}^{2}\right]$$ satisfy $$\sum\limits_{k=1}^K\hat{\beta}_{kj}=0, j = 1,...,p.$$

$p$ is the number of Features, $K$ is the number of classes.

EDIT: I actually think that $\max\limits_{\{\beta_{0k},\beta_{k}\}_{1}^{K}}\left[\sum\limits_{i=1}^{N}logPr(g_{i}| x_i)-\frac{\lambda}{2}\sum\limits_{k=1}^{K}\|\beta_{k}\|_{2}^{2}\right]$ is a Lagrange optimization Problem, so derivating with respect to $x$ and $\lambda$ and then Setting to Zero should do what I Need. But turns out that I'm even getting confused by derivating! $$Pr(g_{i}| x_i) = \dfrac{exp(\beta_{g_i0}+x_i^T\beta_{g_i})}{\sum\limits_{l=1}^Kexp(\beta_{l0}+x_i^T\beta_l)}$$ and I'm not quite sure if I should compute $$\dfrac{\partial f}{\partial x_i}$$ with $i=1,...,N$ or $$\dfrac{\partial f}{\partial x_{ij}}$$ with $i=1,...,N,j=1,...,p$, $f$ being the function inside the square brackets. Well I tried both and I guess it went wrong both times as I couldn't achieve what I'm trying to proove. I didn't forget $\dfrac{\partial f}{\partial \lambda}.$

$\endgroup$
1
$\begingroup$

I think you've got yourself a bit lost here.

Hint #1 - assume you have a set of betas which does not sum to zero, but instead you have $\sum_{k=1}^K\beta_{kj}=c $.

Hint #2 - what happens to the first term $\log [Pr(g_i|x_i)] $ if you add the same constant to each intercept term $\beta_{0k }=\beta_{0k}+d $ ?

$\endgroup$
  • $\begingroup$ Regarding Hint #2: Nothing happens to the first term, $Pr(G=k|X=x)=\dfrac{exp(\beta_{k0} + x^T\beta_k)}{\sum\limits_{l=1}^Kexp(\beta_{l0} + x^T\beta_l)}$ in this case, so the additional d cuts itself out. $\endgroup$ – machinelearner Dec 15 '17 at 11:36
  • $\begingroup$ How can you use this in combination with hint #1? $\endgroup$ – probabilityislogic Dec 15 '17 at 11:47
  • $\begingroup$ Do you want me to regard $\sum\limits_{k=1}^K\beta_{kj}$ as a constant $d$ that's added to $\beta_{0k}$? $\endgroup$ – machinelearner Dec 15 '17 at 19:09
  • $\begingroup$ Does the second term vary if you add a constant $e$ to each term. Is there an optimal choice of what to add? $\endgroup$ – probabilityislogic Dec 15 '17 at 22:47
  • 1
    $\begingroup$ How do you get $e=0$ as the best choice? This is least squares 101... best choice for $e $, take derivative and set to zero. Solve for $e$... $\endgroup$ – probabilityislogic Dec 16 '17 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.