I'm writing a series of blog posts on the basics of machine learning, just for fun, mostly to validate my understanding of Andrew Ng's class. As I'm currently studying generalized linear models (GLMs), my method so far is to generate a small 2D dataset for each regression algo, and apply batch gradient descent on the corresponding error function, to train the parameters. I use Python tools to try to illustrate and interpret the results in an intuitive way.

Following my first post on linear regression:


I've been able to build my logistic regression post in such a way as to emphasize the geometric interpretation of the trained $\theta$ parameters, i.e. show that they correspond to the parameters of the equation of the 2D decision boundary in the general form, $ax + by + c = 0$:


Next I've been trying to do the same with softmax regression (work in progress, not yet posted):


and everything seems fine (i.e. the negative log-likelihood is getting minimized, as well as the classification error, as my notebook graphs show) but I run into difficulties when I try to interpret the $\theta$ parameters in a geometric way, as I did with logistic regression: the resulting decision lines don't make sense (as the last graph of my notebook shows). I have many doubts: does it first make sense to try to interpret those parameters in that way? Or perhaps there's a bug in my training algo? Or something else?

Update (requiring no external reading):

If I train a logistic regression model on 2D data, the resulting three components of $\theta$ can be interpreted as the parameters of the decision line equation, in general form $(\theta_0x + \theta_1y + \theta_3 = 0)$, which might yield, when plotted, something like

enter image description here

If I extend this reasoning to the trained parameters of a 3-class softmax regression, the 9 components should correspond to 3 general form equations. However, when I plot them, as below, they don't seem like decision lines, and I'm wondering if it simply makes sense to interpret them geometrically like that. And if not, is there another intuitive way they can be interpreted?

enter image description here

  • 2
    $\begingroup$ See advice at e.g. meta.stats.stackexchange.com/questions/1479/… on explaining abbreviations. For example, I stop at ML, thinking that means "maximum likelihood", but is that meant here? Presumably you think it's obvious that it means "machine learning", and so it is to some people here. Similarly, please explain GLM, NLL. $\endgroup$ – Nick Cox Oct 9 '13 at 18:07
  • 1
    $\begingroup$ Actually I think I found an answer, I'll write about it soon. $\endgroup$ – cjauvin Oct 10 '13 at 21:21
  • 1
    $\begingroup$ @cjauvin: Is now soon enough? $\endgroup$ – kjetil b halvorsen Oct 9 '17 at 2:11
  • $\begingroup$ @kjetil-b-halvorsen I had written something about it.. cjauvin.blogspot.ca/2013/10/softmax-regression-101.html, but it still wasn't entirely conclusive I fear $\endgroup$ – cjauvin Oct 30 '17 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.