As far as I understand one can in principle model the distribution over a set of $k$ categories using e.g.:
As far as I can tell, both use $k$ parameters to model the distribution over $k$ categories.
- Does the "softmax distribution" have a name?
- Why isn't the softmax discussed more frequently in textbooks?
- If the Dirichlet distribution is a common distribution of choice for categorical outcomes, why do NNs use softmax layers instead of Dirichlet ones?
Almost very textbook that I have encountered and that introduces common distributions (Binomial, Beta, Gaussian, Gamma, Poisson, Dirichlet) never mentions it (e.g. see this list of common distributions in Wikipedia).
Why isn't it presented more commonly in introductory books as type of common distribution? For almost any common distribution, Wikipedia (for example) covers their moments, marginal distributions, entropy, relationship to other known distributions, etc. Why not the same with the softmax distribution?
This is quite perplexing for me given the fact that softmax layers are actually extremely common in neural networks, and the converse is equally true, I have never encountered neural networks using "Dirichlet layers" for modeling categorical outcomes.
What explains this discrepancy? Is it a matter of tradition? Or is there a deeper connection and equivalency between them, and they just happen to go by different names?