For Multinomial Logistic Regression we can define the Loss Function in the following way:

$J(\theta)=\frac{-1}{m}\sum\limits_{i=1}^m\sum\limits_{j=1}^k 1(y^{(i)}=j)\log(\frac{\exp(\theta_j^{T}x^{(i)})}{\sum\limits_{l=1}^k\exp(\theta_l^{T}x^{(i)})})$

When I am trying to find the derivative of this expression with respect to $\theta$, I have:

$J(\theta)=\frac{-1}{m}\sum\limits_{i=1}^m\sum\limits_{j=1}^k 1(y^{(i)}=j)(x^{i}-x^{i}\frac{\sum\limits_{l=1}^k\exp(\theta_l^{T}x_i)}{\sum\limits_{l=1}^k\exp(\theta_l^{T}x_i)})$

However, this expression is totally incorrect because according to this website: http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression I should get:

enter image description here

Nevertheless, I do not understand at all how he obtained this result. Do you know which steps the teacher followed in order to find this result ?

Thank you so much for your help


1 Answer 1

  1. Transform log of fractions into the difference of logs
  2. Note that sum of values of indicator function over all classes equals to 1.
  3. Differentiate it and it should coincide with what is written in the material you are referring to.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.