How to model the correlation between predictions in classification problems In most of deep learning classification problem settings, the predictions $p(y|x)$ are often modeled to be independent, namely,
$$p(y|x) = \prod_{i=1}^{N} p(y_i|x_i)$$
However, this assumption may violate our prior knowledge that the labels could be highly correlated. For example, in face attribute classification, the hair color could be highly correlated to the eye color, and the above model lacks the ability to infer the correlation between predictions, such as $p(y_i, y_j|x)$ or $I(y_i;y_j|x)$.
I would like to know if there is any literature or papers discussing modeling the correlation between predictions given only the dataset $\{x_i, y_i\}_{i=1}^N$. Or, if necessary, given certain additional information is it possible to model the "conditional" correlation $p(y_i, y_j|x)$?
Any input would be greatly appreciated.
 A: Actually it's in evaluation mode at this point of time. 
Early solutions: People used to transfer learning, where once they learnt good model for one task(hairline colour), they use the weights of the initial layers of that model(hairline colour model) to another model for new task(eye colour). Here they use previous model weights for initialisation problem only, which will help in converging the training quickly. And thereafter, both models grow independently from there onwards, still this is one of the predominant solution in production. 
Current state of art solution: Multi head training is getting popular in research, where they share common input and first few hidden layers, then at the end they diverge to their own layers to learn specific tasks, this way when you are learning for the hair line color task the weights learnt (through back propagation) will be helpful for learning eye line color as well and vice versa, together they improve the model since we are back propagating on n correlated tasks. 
The example architecture for your problem would look like below (Note that we can have any type and any number of layers in common stack / specific task stack). Here task 1 is eye line colour, task 2 is hair line colour, and task 3 could be like eyes shape... 
 
