What is the difference between multinomial and ordinal logistic regression?

Question

Can somebody explain it with a simple example!

Answer 1

In the case of the multinomial one has no intrinsic ordering; in contrast in the case of ordinal regression there is an association between the levels. For example if you examine the variable $V1$ that has green, yellow and red as independent levels then $V1$ encodes a multinomial variable. If you have a new variable $V2$ were the levels green, yellow and red represent some sort of increased urgency then they define an ordinal variable.

So how would these variables be encoded:

In the case of an multinomial variable this variable is encoded as an indicator matrix. The arbitrary encoding for green can be [1 0 0], yellow can be [0 1 0] and [0 0 1] for red. In the case of an ordinal variable the encoding is a bit different. If you are in the yellow level you are assumed that you have reached and exceed the green level. Similarly if you are in the red level you have reached the green and the yellow level and now you are in the red level. Therefore the encoding is for the green value would be something like [1 0 0]. For the yellow, [1 1 0] and for red, [1 1 1].

So for the multinomial $V1$ with 7 samples would look something like:

\begin{align} V1 = \left( \begin{array}{ccc} 0 & 0 & 1 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{array} \right) \end{align}

while the ordinal $V2$ with 7 samples would look something like:

\begin{align} V2 = \left( \begin{array}{ccc} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 1 & 0 \end{array} \right) \end{align}

Notice that in both case you are encoding the same information. Namely red, red, yellow, green, green, green, yellow. As you see the difference between the variable $V1$ and $V2$ is obvious now and if you attempt to fit a model in $V1$ that model will almost surely be suboptimal to predict $V2$. The estimation procedures for these models are not vastly different actually. Without going in details both of them rely on numerical optimisation where something called Fisher scoring is extensively used.

Let's look at an actual example now using R. We define a multinomial variable V1 and we then use it to define the ordinal variable V2. True to a real-life setting we only have irrelevant information which is encoded in GarbageInfo. (I use the function multinom from the nnet package to fit the multinomial and the function polr from the MASS package to fit the ordered logit, you can get more information from the documentation of these functions directly.)

set.seed(1234);
N = 100;
V1 = sample(c('green','yellow','red'), N, replace = TRUE)
V2 = ordered(V1, c('green', 'yellow', 'red'))
GarbageInfo = runif(N); # This is used only for illustration purposes
m1 = nnet::multinom(V1 ~ GarbageInfo)
m2 = MASS::polr(V2 ~ GarbageInfo)

We then check the model summaries and notice very interesting.

> summary(m1)
Call: nnet::multinom(formula = V1 ~ GarbageInfo)

Coefficients:
       (Intercept) GarbageInfo
red     -0.6011338  -0.1331142
yellow  -0.3221203  -0.5995860

Std. Errors:
       (Intercept) GarbageInfo
red      0.5164521   0.8448432
yellow   0.4932972   0.8380932
...


> summary(m2)
...
Call: MASS::polr(formula = V2 ~ GarbageInfo)

Coefficients:
              Value Std. Error t value
GarbageInfo -0.2181     0.6431 -0.3391

Intercepts:
             Value   Std. Error t value
green|yellow -0.1541  0.3910    -0.3941
yellow|red    0.9856  0.4045     2.4364
...

While in the case of m1 your intercept is defined for red and yellow against the green baseline, the intercepts for m2 are defined as green|yellow and yellow|red, ie. they are cut-points rather than simple intercepts. Furthermore your GarbageInfo variable coefficient is common in m2 throughout the model in the case of the ordinal regression rather than being estimated independently for each level in the case of the multinomial. That is because you exploit the fact that your data have more information and (in this case) you have an additional degree of freedom. This brings us back to the first sentence of this post: Multinomial and ordinal variables (and their respective) regression procedures are different because they encode different information.