5
$\begingroup$

Copying the definition of multinomial logit regression equations from wikipedia, the probability that the $i$-th observation belongs to class $k$ (out of $K$) is:

$$ \begin{align} \Pr(Y_i=1) &= {\Pr(Y_i=K)}e^{\boldsymbol\beta_1 \cdot \mathbf{X}_i} \\ \Pr(Y_i=2) &= {\Pr(Y_i=K)}e^{\boldsymbol\beta_2 \cdot \mathbf{X}_i} \\ \cdots & \cdots \\ \Pr(Y_i=K-1) &= {\Pr(Y_i=K)}e^{\boldsymbol\beta_{K-1} \cdot \mathbf{X}_i} \\ \end{align} $$

with

$$\Pr(Y_i=K) = 1 - \sum_{k=1}^{K-1}{\Pr(Y_i=K)}e^{\boldsymbol\beta_k \cdot \mathbf{X}_i}$$

and

$$\boldsymbol\beta_k \cdot \mathbf{X}_i = \beta_{0,k} + \beta_{1,k} x_{1,i} + \beta_{2,k} x_{2,i} + \cdots + \beta_{M,k} x_{M,i}$$

for the vector of covariates $(x_{1,i}, x_{2, i}, \dots, x_{M, i})$ associated with the $i$-th observation.

As written, this seems to imply that in the Wikipedia definition the same set of covariates are used to predict class membership probability.

What would happen if the we allowed the set of covariates to be different for each regression equation, i.e.:

$$\Pr(Y_i=k) = {\Pr(Y_i=K)}e^{\boldsymbol\beta_{k} \cdot \mathbf{X}_{k,i}}$$

Is this useful? Would it cause problems in model fitting? Or does the original definition already include this case, whereby $\mathbf{X}_i$ is just the union of all $\mathbf{X}_{k,i}$, with some regression coefficients fixed at zero?

EDIT: This is made in response to Tim's answer which I do not get, because of the following example. Let $Y | X$ be categorically distributed with three classes. Then, the generative model is:

$$\begin{split} \Pr(Y = 3) &= \frac{1}{1 + e^{\alpha x_1} + e^{\beta x_2}}\\ \Pr(Y = 2) &= \frac{e^{\beta x_2}}{1 + e^{\alpha x_1} + e^{\beta x_2}}\\ \Pr(Y = 1) &= \frac{e^{\alpha x_1}}{1 + e^{\alpha x_1} + e^{\beta x_2}} \end{split}$$

Under what kind of regression modelling framework can the original $\alpha$ and $\beta$ coefficients be recovered?

$\endgroup$
  • $\begingroup$ I have a suspicion that what I am asking about is actually the conditional logit model: data.princeton.edu/wws509/notes/c6s3.html, except that page seems to filled with lots of things that I don't find meaningful. $\endgroup$ – Alex Aug 14 '17 at 9:44
  • $\begingroup$ The link is perhaps better expounded here: lehigh.edu/~muy208/teaching/eco416/spring2013/lectures/week08/… $\endgroup$ – Alex Aug 14 '17 at 9:48
  • $\begingroup$ From your description it does not follow that you're talking about conditional logit model. In CL models you also have variables that are shared across the scenarios. $\endgroup$ – Tim Aug 14 '17 at 11:05
3
$\begingroup$

The multinomial regression model is

$$ \Pr(Y = k \mid \mathbf{X}) = \frac{ \exp\big( \beta_k \mathbf{X} \big) }{ 1 + \sum_{i=1}^{K-1} \exp\big( \beta_i \mathbf{X} \big) } \tag{1} $$

What you want to estimate is

$$ \Pr(Y = k \mid \mathbf{X}_k) = \frac{ \exp\big( \beta_k I(Y = k) \mathbf{X}_k \big) }{ 1 + \sum_{i=1}^{K-1} \exp\big( \beta_i I(Y = i) \mathbf{X}_i \big) } \tag{2} $$

So the first consequence to notice, is that you are not estimating the conditional probability of $Y \mid \mathbf{X}$, at least not in the sense that there is a linear dependence of $Y=k \mid X_k$.

Now, since both terms $\alpha_k$ and $\beta_k \mathbf{X}_k $ are unique for each $k$, then any one of them can, as well, be equal to zero and you would still have the model

$$ \Pr(Y = k \mid \mathbf{X}_k) = \frac{ \exp\big( I(Y =k)\cdot \text{something}_{\,k} \big) }{ 1 + \sum_{i=1}^{K-1} \exp\big( I(Y =i)\cdot \text{something}_{\,i} \big) } $$

so you model reduces to

$$ \Pr(Y = k) = \frac{ \exp\big( \alpha_k \big) }{ 1 + \sum_{i=1}^{K-1} \exp\big( \alpha_i \big) } $$

since you are not estimating any conditional probabilities, and for each $k$ the formula needs to return some constant that is equal to $\Pr(Y = k )$.

Saying this otherwise, you would get the same result if $\beta_k I(Y = k) \mathbf{X}_k = \alpha_k$, since $\frac{ \exp( \beta_k I(Y = k) \mathbf{X}_k ) }{ 1 + \sum_{i=1}^{K-1} \beta_i I(Y = i) \mathbf{X}_i } $ needs to evaluate to a single value for each $k$. So $\beta_k I(Y=k) \mathbf{X}_k$ does not let you to estimate any linear dependence of $Y = k \mid \mathbf{X}_k$.

Regarding your example, if you rename $\alpha X_1, \beta X_2$ to $a, b$, then you would clearly see that those values need to be constants, so there is no way how regression parameters would tell you anything meaningful in here. It is as if you had a logistic regression model and used it for data containing only successes (or failures).

$\endgroup$
  • $\begingroup$ Thanks. I do not follow the bit of logic starting at "since both terms αkαk and βkXkβkXk are unique for each k, ..." and how that leads to the model reducing to the inverse logit of a constant. I also do not understand why you say I am not "estimating the conditional probability", as I think considering $Y | \mathbf{X}$ where $\mathbf{X}$ is the union of all $\mathbf{X}_k$ is valid? $\endgroup$ – Alex Aug 14 '17 at 8:24
  • $\begingroup$ @Alex for $Y=a$ you take $\beta_a X_a$, for $Y=b$ you take $\beta_b X_b$ etc. The formulas are equal to zero for all other cases. So you end up with single possible solution of $\alpha_k + \beta_k X_k$ for each $k$, it doesn't make sense to estimate $\alpha_a + \beta_b X_b$ since you do not have $X_b$, so you are not conditioning $Y = a$ on $X_b$. For $Y = a$ you have $X_b = \emptyset$. $\endgroup$ – Tim Aug 14 '17 at 8:30
  • $\begingroup$ Ok, I will have to think about what happens when $\mathbf{X}_k$ are pairwise distinct. What about in the intermediate case where they share elements though? $\endgroup$ – Alex Aug 14 '17 at 8:33
  • $\begingroup$ @Alex "everything different" is the extreme case that reduces to estimating only the intercepts. If you had shared variables, then you model the interaction $\text{dummy}_{\,k} \cdot X_k$ (same scenario if you, e.g. used dummies for missing values). $\endgroup$ – Tim Aug 14 '17 at 8:38
  • 1
    $\begingroup$ @Alex I made some edits to make it more clear (I started with writing it in terms of logistic regression to make it simpler, but it made it more confusing). Plus, I re-writed the answer partially, so check if it is more clear nor. $\endgroup$ – Tim Aug 14 '17 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.