Do neural networks implicitly learn correlation between outputs when performing multi-label classification or multi-output regression? If not, is there a way for them to do so? I have seen that there is work on using RNN's to capture label interdependence for multi-label image classification and captioning. However, I do not believe that my problem is particularly well suited for RNNs.


The question is, if the likelihood function you specified does so. E.g. if you model 2 variables $y_1, y_2$, you will often use a likelihood function of the form

$$ p(y_1, y_2 | x) = p(y_1|x) p(y_2|x). $$

For example, this is the same as summing up the log-likelihoods of two Bernoulli variables if you do two classificiations simultaneously for a loss function. This will not capture correlations, as the likelihood function assumes independency.

If you want to capture correlations, you need to use a different likelihood function. Two ways of doing so are a) formulate the problem in an autoregressive fashion or b) using additional stochastic variables. In the former case, you use a likelihood function of the form

$$p(y_1, y_2 | x) = p(y_1|x) p(y_2|y_1, x).$$

Note the $y_1$ in the condition for $y_2$. A starting point on how to do this is [1]. The latter is accounts to a graphical model of the form

$$p(y_1, y_2 | x) = \int p(y_1|x, z) p(y_2|x, z) p(z|x) dz.$$

Check out [2] for a description on how to learn these beasts.

[1] Uria, Benigno, et al. "Neural Autoregressive Distribution Estimation." Journal of Machine Learning Research 17.205 (2016): 1-37.

[2] Tang, Yichuan, and Ruslan R. Salakhutdinov. "Learning stochastic feedforward neural networks." Advances in Neural Information Processing Systems. 2013.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.