What is the statistical model for a multi-label problem?

Question

In a setting with a binary $y$ like dog/cat, a reasonable statistical model is to posit that the probability parameter $p$ of a $\text{Binomial}(1, 0)$ distribution is some function $f$ of features $X$. This leads to many common machine learning approaches like logistic regression and neural networks.

In a setting with multiple classes in $y$, such as dog/cat/horse, a reasonable statistical model is that the multiple probability values $\vec p$ of a $\text{Multinomial}(1, \vec p)$ distribution is some function of features $X$. Much like in the binary setting, this leads to many common approaches like multinomial logistic regression and various forms of deep learning (e.g., convolutional neural networks for MNIST handwritten digits).

In a multi-label setting, such as identifying if a photograph contains a dog, a cat, or both, what would be the statistical model?

(To a large extent, I think I mean the formal likelihood, regardless of the functional form of expressing the probability of class membership as depending on the features, but I want to leave it a bit vague to allow for an answer that I’m thinking about it wrong by thinking of statistical likelihood.)

EDIT

The comments have clarified that an Ising likelihood works. With that being the case, how do the prior probabilities of the classes come into the picture? For instance, in a logistic regression, if I have $99$ $0$s for every $1$, I expect low probabilities of $1$ unless the features are extremely informative. In a multi-label setting, it seems like the prior probability of each class would be the ratio of that class to all possible alternatives, which I would consider to be zero: out of every possible sight there is to see, the probability of seeing a dog ought to be tiny or even zero (and the fact that we’re on Earth (for now) and near dogs is what allows us to see dogs with frequency).

It seems like something like this fails for multi-label classification.

Answer 1

Let's generalise your example by supposing that we have $K$ types of animal/object. As you correctly point out, if we are dealing with observations where these types are mutually exclusive then we would have $K$ possible outcomes and so we would use some kind of multinomial model with $K$ categories. However, if these animals/objects are not mutually exclusive then we would now have $2^K$ possible outcomes so we could now use some kind of multinomial model with $2^K$ categories. In principle this latter case is not really all that different from the former case, and the same types of models can be used. In your example of a dog/cat/horse model, you have $K=3$ so there would be $2^K=8$ possible outcomes.

Note that this type of model does not impose any assumed structure on the combinations of animals/objects in the outcome variable; it is appropriate if $K$ is not too large. If $K$ is large enough that $2^K$ is too large to deal with effectively then we would probably impose more structure on the model to specify some kind of parametric relationship between the animals/objects occurring in an outcome. At that point the two forms of model would become different.

Answer 2

You can think of the multinomial (aka multinoulli) distribution used in multinomial classification as a multivariate discrete distribution obtained in two steps:

1. assign a Bernoulli marginal distribution to each of the labels;
2. use a very specific copula function to model the dependence between the marginals; the copula is such that the Bernoullis are perfectly dependent (if one of them takes value 1, all the others take value 0).

In a multi-label problem, you change the copula because you drop the assumption of perfect dependence (two or more Bernoullis are allowed to simultaneously take value 1).

The problem is that you need to decide which copula to use. For example, you could use the product copula (all the Bernoullis are independent), similarly to what is done in naive Bayes models.

There are several obstacles to choosing / specifying an appropriate copula function:

1. if the number of labels ($K$) is large, you incur in the curse of dimensionality; roughly speaking, you need to model $K^2$ correlations between the Bernoullis, as mentioned in Ben's answer;
2. there are infinitely many possible dependence structures you could choose;
3. it's unclear whether the dependence structure needs to be parametrized separately (before training the model) or estimated while training the model;
4. the copula of a discrete random vector is not fully identifiable, which can cause serious inconsistencies.

That said, I think that the easier way to go is to use the product copula (make the Bernoullis independent), which is equivalent to training $K$ separate binomial classification models (one for each label), but with parameter sharing in all the layers but the last one.