Multinomial Regression: How to understand when no "category" is evident?

Question

I am having trouble understanding multinomial regression, aka softmax regression, multinomial logit regression, discrete-choice modeling, etc.

My main confusion comes from thinking about two separate cases of multinomial regression - perhaps one is not in this case.

From what I understand, multinomial regression can be employed to model the probability of an event (i.e. choice) as a function of the characteristics associated with that event and the characteristics of the other possible, alternative events.

So, if our event is "choosing" alternative j and there are multiple individuals, the probability of individual i choosing alternative j is given by:

But I commonly see this presented in terms of alternatives belonging to some set of categories that is consistent across cases. For example, the probability of choosing to vote for person A from a group of people {A, B, C, D}. This is referred to as the choice set.

The issue I am having is this same technique can be applied when there is no way to reasonably group the alternatives into categories, and when there are multiple observations of an individual making a choice, with different choice sets.

For example, an individual at location 1 could have chosen 2, 3, or 4, each of which have their own set of predictor covariates. Further, that individual was also observed at location 5, where they could have chosen 6, 7, or, to add another layer of my confusion, 4, which is also part of a different choice set.

I'm having trouble understanding whether or not these are really separate examples, or how the same model applies to them both. Perhaps its because I keep thinking about the multinomial distribution as generating probabilities for a "fixed" group of categories? I just can't pinpoint the exact source of confusion - any comments would be greatly appreciated.

Answer 1

As a caveat, I don't have extensive experience with this models, but I'll offer a few thoughts nonetheless.

A traditional multinomial logit regression makes assumptions about the data being modeled. For example, the items that populate the choice set must be discrete. Thus, if we are modeling votes, using a multinomial logit regression, you would assume that a person cannot vote for two people. This also means that data points are assumed to be dependent -- you do not have multiple measurements or outcomes for any one individual/case.

This situation is more or less what you describe in your first example.

In your second example, clearly the assumptions made by a multinomial logit regression are violated -- so you are right to be concerned! However, all that one would need to do is account for these violations.

In your second example, you still have a finite set of choices. {1,2,3,4,5,6}. But, a few things have changed. First, the probability of choice depends on location, which needs to be built into the model. E.g. we need to represent the fact that:

$Pr(Y = 6| Location = 1) = 0$

Also, if it is the case in your data that each person could potentially make multiple choices, this needs to be reflected, too. The problem is that you have to assume that the set of choices made by a given person will be correlated, due to any number of unmeasured variables. This within-person correlation violates the assumption of independence and it is exactly the same problem you run into repeated measures experimental designs.

While there are multiple ways you could approach modeling this example, one would be a hierarchical multinomial regression model. In this model, each choice datapoint would be associated with a person, the location at which the choice was made, and whatever other covariates you wanted. Thus, if person A made multiple choices, this would be represented in the data structure.

Further, you would model the effect of person on choice. Remember, because choices are nested (and potentially repeated) within people, we have to assumed that choices are correlated within people, which means that everything aside, the probability that of choosing X depends on the person making the choice and therefore might be correlated with the probability of choosing Y.

Also, since you specify that choice 4 can be made at either location, you might also want to model a random effect for choice. In this case, you would be indicating that the probability of choosing 4 varies as a function of location AND person.

I haven't read it, but a google search pulled up this chapter that might be relevant to your questions: http://faculty.education.illinois.edu/cja/homepage/Discrete_Model_proofs_marked2.pdf.

Just to summarize, you are correct that both examples you gave depend on a multinomial distribution. They are, in spirit, the same. However, an key difference is that the mechanisms that generated the data in the second example is much more complicated and, more importantly, it violates the assumptions made by standard multinomial regression models. This does not change the essence of the problem, but it does require that you find a way to account for these violations in your model of the data.