(For reference, i am using Kevin P Murphy's Book "Machine Learning: A Probabilistic Perspective" and implementing with MATLAN - without any toolboxes)

I have a dataset with 392 samples (rows), each sample has 8 features (columns), one of which defines the class (i.e. column 1 of features is divided into three equal bins which define the three classes - low, medium, and high).

I am having a really hard time understanding how to create a logistic regression model to classify a sample into one of these three classes.

I just finished learning and making a linear regression model where I learned aboutboth the Ordinary Least Squares (Closed form) solution for the weight vector, and also Gradient Descent (Open Form) solution. But i never implemented gradient descent because my data was fitted perfectly fine with the OLS solution for weight vector.

I am extremely confused how to create a weight vector for Logistic regression, I understand that it requires use of Gradient Descent because there is no closed form solution. I also read about the Newton method for calculating the weights but I don't understand it at all.

And after you use these methods to calculate weights how do you apply the weights to the sample data? In Linear regression it was simply because you simply multiplied the weights by the features (and higher order features for higher order linear regression), but is it the same in logistic regression?

Moreover my understanding so far is that this model only works for binary classification, so how would i do it for three classes?

Basically my question boils down to this:

How exactly do you find the weight vector for logistic regression (using either gradient descent or newtons method, whichever is easier) and how do you apply the weight vector to the sample to get a classification out of it for 3 classes (not just two).


1 Answer 1


Firstly, it is worth pointing out that what you're looking to do is called multinomial logistic regression in the statistics literature.

Since you're reading a machine learning book, however, I'm going to suggest an alternate source: read the Neural Networks chapters from David Mackay's textbook, which you can view online for free (but not print) here: http://www.inference.phy.cam.ac.uk/itprnn/book.pdf.

Why neural networks? Because with just input and output layers (no hidden layers) they are equivalent to logistic regression. The chapters take you through the binary case, what the objective function is, what the gradient and update rule for the weight vector is, and then in Chapter 44 there's material on 'Multi-class classification networks' which do exactly what you want. The treatment is quite practical which might be useful for someone without a lot of background. You can then go back and read about (multinomial) logistic regression in a stats textbook to get a more formal appreciation for it if you want to.

  • $\begingroup$ The "Elements of Statistical Learning" book has good explanation of multinomial logistic regression and how to fit that model in Chapter 4. $\endgroup$
    – purple51
    Commented Oct 15, 2014 at 10:24
  • $\begingroup$ thanks but my goal is to learn how to implement multinomial logistic regression. Neural networks is a topic i plan to learn later. $\endgroup$ Commented Oct 15, 2014 at 23:46
  • $\begingroup$ As I said---a no-hidden-layer neural net with softmax output is multinomial logistic regression $\endgroup$ Commented Oct 16, 2014 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.