Do deep learning algorithms run into trouble when tasked with classifying high dimensional input into one of many categories? By many I mean thousands or millions. If it does, how could one deal with this problem? Any references?

  • $\begingroup$ by deep learning do you mean neural networks? $\endgroup$ – Franck Dernoncourt Dec 30 '16 at 17:25
  • $\begingroup$ @FranckDernoncourt yeah, sorry I'm not up to speed on the jargon with these things $\endgroup$ – Taylor Dec 30 '16 at 17:35
  • 1
    $\begingroup$ In science, when categorizing with millions of possible categories, usually some hierarchcal system is built, like in biological systematics. There will be two cases: Building such a system, or cateforizing into a known system. Which case is yours? $\endgroup$ – kjetil b halvorsen Dec 30 '16 at 18:47
  • 1
    $\begingroup$ @kjetilbhalvorsen thanks that's helpful. I don't really have one, I am just being curious at the moment. I know the output of say a CNN will give you a vector of probabilities over your categories, so I am curious if and why things become hard to discern when looking at examples besides this MNIST dataset. The more bins you divide up $1$ into, the smaller the differences in their volume. just wondering $\endgroup$ – Taylor Dec 30 '16 at 19:02

If it does, how could one deal with this problem? Any references?

You can use hierarchical softmax, importance sampling, noise contrastive estimation, or negative sampling: they are commonly used in language modeling, for example.


| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.