I'm trying to solve an image classification problem that involves a very large number of classes. Each image actually has two labels associated with it that we can think of as a coarse and a granular label (so there are many more granular labels than coarse labels and any given example has one of each, etc.).
I have a deep neural network classifier that operates on the coarse label and it is extremely successful. I have a classifier with similar network topology that operates on the granular label and it is less successful (not surprisingly).
One thing I'd like to try is having the network predict both labels at the same time. How would one approach this problem? I'm considering the naive thing and just having two different softmax layers at the end and computing a loss for each of them, and then summing the loss. Is that sensible? One big drawback is that if the predicted granular label is invalid for the coarse label, that isn't naturally reflected in the loss. Are there any papers on this sort of material, etc.?
Obviously, another thing I could try would be to have a two stage model - run the coarse classifier first and then run a specialized model to do the fine classification, but that would involve a lot of re-processing my data, and would probably be a bit bulky here as well.