0
$\begingroup$

As far as I understand one can in principle model the distribution over a set of $k$ categories using e.g.:

As far as I can tell, both use $k$ parameters to model the distribution over $k$ categories.

My questions:

  • Does the "softmax distribution" have a name?
  • Why isn't the softmax discussed more frequently in textbooks?
  • If the Dirichlet distribution is a common distribution of choice for categorical outcomes, why do NNs use softmax layers instead of Dirichlet ones?

Context

Almost very textbook that I have encountered and that introduces common distributions (Binomial, Beta, Gaussian, Gamma, Poisson, Dirichlet) never mentions it (e.g. see this list of common distributions in Wikipedia).

Why isn't it presented more commonly in introductory books as type of common distribution? For almost any common distribution, Wikipedia (for example) covers their moments, marginal distributions, entropy, relationship to other known distributions, etc. Why not the same with the softmax distribution?

This is quite perplexing for me given the fact that softmax layers are actually extremely common in neural networks, and the converse is equally true, I have never encountered neural networks using "Dirichlet layers" for modeling categorical outcomes.

What explains this discrepancy? Is it a matter of tradition? Or is there a deeper connection and equivalency between them, and they just happen to go by different names?

$\endgroup$
1
$\begingroup$

The softmax function is a function. For a fixed input, you get a fixed output, just as for any other function. Someone might have invented a softmax distribution in some context, but that's not the meaning of "softmax function," as described in your post, or commonly used in neural networks.

The Dirichlet distribution is a distribution. Drawing from a Dirichlet distribution with some parameters will almost surely give a different result each time. The Dirichlet distribution is not a distribution over categories, it's a distribution over probability vectors.

The commonality that you've found between the softmax function and the Dirichlet distribution is that the results (function outputs, random draws) are always probability vecotrs: the vectors are non-negative, and sum to 1.

The normal distribution is a probability distribution which assigns positive probability to any real number. A linear function yields real number for any input. But you wouldn't conflate the two, because they're clearly different things in important ways. Likewise, the softmax function and Dirichlet distribution are distinct.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks Sycorax! But once you get a vector of probabilities, sampling from that vector is independent of how we get them, isn't it? In other words, I could create a distribution from the Softmax distribution and sample from it in the same way I would sample from a Dirichlet distribution, no? $\endgroup$ – Josh Aug 3 at 0:56
  • 1
    $\begingroup$ @Josh You're using "sampling" in a strange sense. You could take a sample from a Dirichlet and the results of a softmax and use that set of probabilities as hyperparameters for something else. But you're not sampling from a softmax. It's not a distribution. $\endgroup$ – Wayne Aug 3 at 1:16
  • 1
    $\begingroup$ @Josh Sampling from a Dirchlet distribution yields a vector of probabilities. It's not clear what you mean by sampling from a "softmax distribution" because softmax isn't a distribution -- or, at least, you haven't described how a softmax distribution is related to a softmax function. Perhaps you're conflating a distribution over several categories with a distribution over probability vectors? As contrast, the beta distribution is a distribution over probability, and the binomial distribution is binary, but the beta-binomial is its own beast. $\endgroup$ – Sycorax Aug 3 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.