14
$\begingroup$

Can somebody explain it with a simple example!

$\endgroup$
1
  • $\begingroup$ Can someone explain, from a probabilistic standpoint and the proportional odds assumptions, why does it make sense to simply encode the matrix differently? Or is this specific to the R package? I don't see how simply recoding the matrix can change the problem from a multinominal to ordinal, especially given that for the latter that is a cutoff point $c$, where $P(Y=1|X) = P(Y^{*}>c)$, where $Y^{*}$ would be a latent variable; in the former this is not exactly the case, $P(Y=1|X) = P(Y^{*}_{k}>Y^{*}_{k+1}, Y^{*}_{k}>Y^{*}_{k+2} ... )$ where $k$ is the class. $\endgroup$
    – deblue
    Nov 15, 2023 at 11:20

1 Answer 1

22
$\begingroup$

In the case of the multinomial one has no intrinsic ordering; in contrast in the case of ordinal regression there is an association between the levels. For example if you examine the variable $V1$ that has green, yellow and red as independent levels then $V1$ encodes a multinomial variable. If you have a new variable $V2$ were the levels green, yellow and red represent some sort of increased urgency then they define an ordinal variable.

So how would these variables be encoded:

In the case of an multinomial variable this variable is encoded as an indicator matrix. The arbitrary encoding for green can be [1 0 0], yellow can be [0 1 0] and [0 0 1] for red. In the case of an ordinal variable the encoding is a bit different. If you are in the yellow level you are assumed that you have reached and exceed the green level. Similarly if you are in the red level you have reached the green and the yellow level and now you are in the red level. Therefore the encoding is for the green value would be something like [1 0 0]. For the yellow, [1 1 0] and for red, [1 1 1].

So for the multinomial $V1$ with 7 samples would look something like:

\begin{align} V1 = \left( \begin{array}{ccc} 0 & 0 & 1 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 0 & 1 & 0 \end{array} \right) \end{align}

while the ordinal $V2$ with 7 samples would look something like:

\begin{align} V2 = \left( \begin{array}{ccc} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 0 & 0 \\ 1 & 1 & 0 \end{array} \right) \end{align}

Notice that in both case you are encoding the same information. Namely red, red, yellow, green, green, green, yellow. As you see the difference between the variable $V1$ and $V2$ is obvious now and if you attempt to fit a model in $V1$ that model will almost surely be suboptimal to predict $V2$. The estimation procedures for these models are not vastly different actually. Without going in details both of them rely on numerical optimisation where something called Fisher scoring is extensively used.

Let's look at an actual example now using R. We define a multinomial variable V1 and we then use it to define the ordinal variable V2. True to a real-life setting we only have irrelevant information which is encoded in GarbageInfo. (I use the function multinom from the nnet package to fit the multinomial and the function polr from the MASS package to fit the ordered logit, you can get more information from the documentation of these functions directly.)

set.seed(1234);
N = 100;
V1 = sample(c('green','yellow','red'), N, replace = TRUE)
V2 = ordered(V1, c('green', 'yellow', 'red'))
GarbageInfo = runif(N); # This is used only for illustration purposes
m1 = nnet::multinom(V1 ~ GarbageInfo)
m2 = MASS::polr(V2 ~ GarbageInfo)

We then check the model summaries and notice very interesting.

> summary(m1)
Call: nnet::multinom(formula = V1 ~ GarbageInfo)

Coefficients:
       (Intercept) GarbageInfo
red     -0.6011338  -0.1331142
yellow  -0.3221203  -0.5995860

Std. Errors:
       (Intercept) GarbageInfo
red      0.5164521   0.8448432
yellow   0.4932972   0.8380932
...


> summary(m2)
...
Call: MASS::polr(formula = V2 ~ GarbageInfo)

Coefficients:
              Value Std. Error t value
GarbageInfo -0.2181     0.6431 -0.3391

Intercepts:
             Value   Std. Error t value
green|yellow -0.1541  0.3910    -0.3941
yellow|red    0.9856  0.4045     2.4364
...

While in the case of m1 your intercepts are defined for red and yellow against the green baseline, the intercepts for m2 are defined as green|yellow and yellow|red, ie. they are cut-points rather than simple intercepts. Furthermore your GarbageInfo variable coefficient is common in m2 throughout the model in the case of the ordinal regression rather than being estimated independently for each level in the case of the multinomial. That is because you exploit the fact that your data have more information and (in this case) you have an additional degree of freedom. This brings us back to the first sentence of this post: Multinomial and ordinal variables (and their respective) regression procedures are different because they encode different information.

$\endgroup$
2
  • $\begingroup$ Would you draw a parallel between this and embeddings produced by a neural network? With the added benefit that an embedding would capture more precisely (making generous assumptions on the training data and goal of course) the relationship between the variables? If not, then what is the fundamental distinction? $\endgroup$
    – Leo
    Dec 4, 2023 at 0:12
  • $\begingroup$ It is quite open-ended to me what is meant by "embeddings produced by a neural network" as there are a lot of way of producing "an" embedding (computational constraints, required mathematical properties, data availability, etc.) To name an obvious point, in the encoding described above there is a clear user-defined additive interpretation, a standard "embeddings produced by a neural network" (let's say via an auto-encoder) doesn't guarantee that interpretation. Please feel free to start a new question about this! :) $\endgroup$
    – usεr11852
    Dec 4, 2023 at 0:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.