Do neural networks implicitly learn correlation between outputs when performing multi-label classification or multi-output regression? If not, is there a way for them to do so? I have seen that there is work on using RNN's to capture label interdependence for multi-label image classification and captioning. However, I do not believe that my problem is particularly well suited for RNNs.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
The question is, if the likelihood function you specified does so. E.g. if you model 2 variables $y_1, y_2$, you will often use a likelihood function of the form
$$ p(y_1, y_2 | x) = p(y_1|x) p(y_2|x). $$
For example, this is the same as summing up the log-likelihoods of two Bernoulli variables if you do two classificiations simultaneously for a loss function. This will not capture correlations, as the likelihood function assumes independency.
If you want to capture correlations, you need to use a different likelihood function. Two ways of doing so are a) formulate the problem in an autoregressive fashion or b) using additional stochastic variables. In the former case, you use a likelihood function of the form
$$p(y_1, y_2 | x) = p(y_1|x) p(y_2|y_1, x).$$
Note the $y_1$ in the condition for $y_2$. A starting point on how to do this is [1]. The latter is accounts to a graphical model of the form
$$p(y_1, y_2 | x) = \int p(y_1|x, z) p(y_2|x, z) p(z|x) dz.$$
Check out [2] for a description on how to learn these beasts.
[1] Uria, Benigno, et al. "Neural Autoregressive Distribution Estimation." Journal of Machine Learning Research 17.205 (2016): 1-37.
[2] Tang, Yichuan, and Ruslan R. Salakhutdinov. "Learning stochastic feedforward neural networks." Advances in Neural Information Processing Systems. 2013.
