My question pertains to section 2, called "Multi-class Logistic regression", of this pdf, especially the update rules. (The entire section is only a couple of paragraphs.)

Everything seems to make sense, but I don't see how one is supposed to implement the gradient ascent algorithm.

  • What is $y_i^k$? What are its dimensions?
  • What is the dimension of the weights?
  • What is $x_i$? Say you have a matrix where each row is an observation and each column is a feature. What would $x_i$ be? Is it a row?
  • Is $p(k|x_i)$ the same thing as $p(y = k|x)$ for a particular observation?

Side note: I'm trying to implement a simple multi-class logistic regression in MATLAB.

  • 1
    $\begingroup$ Be wary of any field that invents new names for old ideas or that claims that logistic regression is a classifer. The proper terms for what you are seeking are polytomous logistic regression or multinomial logistic regression. $\endgroup$ – Frank Harrell Jun 6 '14 at 14:43

Suppose $Y$ is a multiclass output of dim $N \times 1$, i.e., the $i^{\rm th}$ row of $Y$, denoted $y_i$ can take values in $\{1,2,\ldots,K\}$. Since the author is using 1-of-K coding, $y_i^k = 1$ if $y_i = k$ and $0$ otherwise.

You also have feature matrix $X$ dim $N \times P$. The weight matrix ${\bf w_k}$ will be of dimension $(P + 1) \times 1$ where the $+ 1$ is for an intercept and will correspond to the $k^{\rm th}$ regression $\rightarrow p(y = k | x)$. Based on the notes, $x_i$ is the $i^{\rm th}$ row of $X$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.