Multinomial logit models with nested categories in DV

Question

I've wanted to ask for some help with multinomial logit models. I'm investigating how the prior probability of song lyrics affects what lyrics we actually hear.

In my experiment there are 30 songs (we take only one line from each song), and each song is an independent trial. On each trial, participants listen to a line from the song, and then they are shown 4 different song lyrics and they are asked to select which one is the real line from the song (one response is correct, the rest are false but sound similar). Separately, we used a large language model to obtain the prior probability of every line (real and fake).

I'm struggling to model this because each of the 30 songs has 4 options that participants could choose from; but option 1 from song 1 is independent from option 1 from song 2. Also, we should account for song-level intercepts, because the mean prior probability of the four lines from each song can differ wildly across songs.

I was going with something like the following, where choice_id is a category with 1 to 4*30 levels (four options times 30 songs, total of 120 possible lines), prior is a continuous value for the prior probability, and song is the grouping level for each song (i.e., associated with a group of 4 choice_id):

brm(choice_id ~ prior + (1|song), family=categorical)

But it takes a lot of time to fit and I'm getting a coefficient for each option of the 120 choice options, and I'm not interested in that: just in whether increase in prior probability increases the probability of an option being selected amongst the four options that were presented at each trial.

Thank you so much for your help!

---

Edit: Thanks so much for the response! I see how I'm being unclear and misguided at points.

The research question is whether prior probability predicts what we hear. Sometimes, the real song lyric is less probable than some alternative. For this reason, I don't care if participants choose the real lyrics or not (there is no "wrong"). In a simplified sense, I just want to see if the relative prior of a line (relative prior to the other three lines presented at the same time) predicts how often it will be selected as what people hear. I could transform the entire dataset to counts per line in each song and normalize the prior probability for each of the four options within a song, but this seems like it reduces the amount of information within the dataset (e.g., participant-level variation that I'd like to capture with random effects). Does this make any sense? Thank you for your help :)

Answer 1

A multinomial logit model is typically used when the outcome has more than two possibilities and those possibilities are the same for every observation. You have repeated options for each song. However, these options are unique for each song such that options 1-4 for song 1 are different than option 1-4 for song 2. So right away, I am not sure that multinomial logit is what you want.

At the same time, I don't really understand your research question.

I'm investigating how the prior probability of song lyrics affects what lyrics we actually hear.

How do you operationalize the "lyrics we actually hear"? Is it that one of the choices for each song is the correct lyric that the original artists used? If so, you could re-code the choices such that a choice takes on a value of 0 if that is not the lyric the artist used or 1 if it is the lyric the artist used. Then, you could analyze this using a repeated measures (mixed effects) logit model.

This logit version of the model addresses the question of whether prior is associated with respondents choosing the correct lyric. I don't know if that is what you want, but it is at least is a clearer research question. If it is not what you want, then please provide an operational definition of "what lyrics we actually hear."

This statement is also confusing:

Also, we should account for song-level intercepts, because the mean prior probability of the four lines from each song can differ wildly across songs

In terms of the song-level intercepts, they help to account for the between song differences in the outcome, not the predictors.

---

Edit based on new information

I believe you have described the best approach for moving forward in your newest addition to your original question. It is still a multilevel (mixed) effects model because you have multiple outcomes per song, but what you are analyzing is the counts of how often the line was chosen. This gives you a common metric for the dependent variable. I cannot tell from your description how you get to a common metric from the individual respondent data unless you code the individual observations as 0/1 (not chosen/chosen). That would necessitate a binomial rather than a multinomial logit model.