I am doing a college classification project, in which I am required to classify some handwritten digits. Assume that my input is a N*D where D is the number of features in each input sample and I need to classify each sample into "K" classes. I am using multiclass logisitic regression and in my training phase I am trying to find my weighted matrix(D*K) with gradient descent algorithm.

I have read so many articles about this algorithm and in many of them it is suggested that we need to choose the weighted matrix at which our algorithm converges. I am assuming that here the word "converges" means where all the entries in our gradient matrix will become '0'. I have tried around 1000 iterations of this algorithm and with each iteration my error is going down but my gradient matrix is not becoming 'zero'.

So can someone tell me why I am getting this behaviour or getting a perfect '0' for all of the elements in the gradient matrix is impossible?


First, multiclass logistic regression is a linear classifier, are your inputs linearly separable? If they're handwritten digits I highly doubt it.

Next, multiclass logistic regression uses the softmax function to assign class probabilities,

$$ p(Y = i|X = x) = \frac{\exp(f_i(x))}{\sum_{j=1}^n \exp(f_j(x))}, $$

and normally you will use a $1$-of-$k$ binary encoding for your outputs. However since the output of the softmax only matches this encoding asymptotically as $f_i(x) \longrightarrow \pm\infty$, you'll always have some error and thus your gradient will never be 0.

There are probably many ways you can assess convergence but here are some ideas,

  • Use the value of the $0$-$1$ loss to assess convergence rather than whatever differentiable loss you're using for gradient descent.
  • Stop training when your parameter changes are smaller than some threshold.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.