Do deep learning algorithms run into trouble when tasked with classifying high dimensional input into one of many categories? By many I mean thousands or millions. If it does, how could one deal with this problem? Any references?
If it does, how could one deal with this problem? Any references?
You can use hierarchical softmax, importance sampling, noise contrastive estimation, or negative sampling: they are commonly used in language modeling, for example.
- Why is hierarchical softmax better for infrequent words, while negative sampling is better for frequent words? (The answer gives a brief overview of what hierarchical softmax and negative sampling are)
- http://www.deeplearningbook.org/ section "12.4.3 High-Dimensional Outputs": presents hierarchical softmax, importance sampling, noise contrastive estimation, and negative sampling.
- Dyer, Chris. "Notes on Noise Contrastive Estimation and Negative Sampling." arXiv preprint arXiv:1410.8251 (2014). https://arxiv.org/abs/1410.8251