Can somebody explain it with a simple example!
In the case of the multinomial one has no intrinsic ordering; in contrast in the case of ordinal regression there is an association between the levels.
For example if you examine the variable $V1$ that has green
, yellow
and red
as independent levels then $V1$ encodes a multinomial variable.
If you have a new variable $V2$ were the levels green
, yellow
and red
represent some sort of increased urgency then they define an ordinal variable.
So how would these variables be encoded:
In the case of an multinomial variable this variable is encoded as an indicator matrix. The arbitrary encoding for green
can be [1 0 0]
, yellow
can be [0 1 0]
and [0 0 1]
for red.
In the case of an ordinal variable the encoding is a bit different. If you are in the yellow
level you are assumed that you have reached and exceed the green
level. Similarly if you are in the red
level you have reached the green
and the yellow
level and now you are in the red
level. Therefore the encoding is for the green
value would be something like [1 0 0]
. For the yellow
, [1 1 0]
and for red
, [1 1 1]
.
So for the multinomial $V1$ with 7 samples would look something like:
\begin{align} V1 = \left( \begin{array}{ccc} 0 & 0 & 1 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{array} \right) \end{align}
while the ordinal $V2$ with 7 samples would look something like:
\begin{align} V2 = \left( \begin{array}{ccc} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 1 & 0 \end{array} \right) \end{align}
Notice that in both case you are encoding the same information. Namely red
, red
, yellow
, green
, green
, green
, yellow
. As you see the difference between the variable $V1$ and $V2$ is obvious now and if you attempt to fit a model in $V1$ that model will almost surely be suboptimal to predict $V2$. The estimation procedures for these models are not vastly different actually. Without going in details both of them rely on numerical optimisation where something called Fisher scoring is extensively used.
Let's look at an actual example now using R. We define a multinomial variable V1
and we then use it to define the ordinal variable V2
. True to a real-life setting we only have irrelevant information which is encoded in GarbageInfo
.
(I use the function multinom
from the nnet
package to fit the multinomial and the function polr
from the MASS
package to fit the ordered logit, you can get more information from the documentation of these functions directly.)
set.seed(1234);
N = 100;
V1 = sample(c('green','yellow','red'), N, replace = TRUE)
V2 = ordered(V1, c('green', 'yellow', 'red'))
GarbageInfo = runif(N); # This is used only for illustration purposes
m1 = nnet::multinom(V1 ~ GarbageInfo)
m2 = MASS::polr(V2 ~ GarbageInfo)
We then check the model summaries and notice very interesting.
> summary(m1)
Call: nnet::multinom(formula = V1 ~ GarbageInfo)
Coefficients:
(Intercept) GarbageInfo
red -0.6011338 -0.1331142
yellow -0.3221203 -0.5995860
Std. Errors:
(Intercept) GarbageInfo
red 0.5164521 0.8448432
yellow 0.4932972 0.8380932
...
> summary(m2)
...
Call: MASS::polr(formula = V2 ~ GarbageInfo)
Coefficients:
Value Std. Error t value
GarbageInfo -0.2181 0.6431 -0.3391
Intercepts:
Value Std. Error t value
green|yellow -0.1541 0.3910 -0.3941
yellow|red 0.9856 0.4045 2.4364
...
While in the case of m1
your intercept is defined for red
and yellow
against the green
baseline, the intercepts for m2
are defined as green|yellow
and yellow|red
, ie. they are cut-points rather than simple intercepts. Furthermore your GarbageInfo
variable coefficient is common in m2
throughout the model in the case of the ordinal regression rather than being estimated independently for each level in the case of the multinomial. That is because you exploit the fact that your data have more information and (in this case) you have an additional degree of freedom. This brings us back to the first sentence of this post:
Multinomial and ordinal variables (and their respective) regression procedures are different because they encode different information.