# Is there a SoftMax distribution with confidence parameters?

This is a softmax probability distribution:
$$P(i| w_1, w_2, \ldots, w_n) = \frac{exp(w_i)}{\sum_{i=1}^n exp(w_i)}.$$

It known also as Boltzmann distribution. It is used in generalized Bradley-Terry model and in multinomial logistic regression. There are efficient minorization-maximization algorithms for infering $\vec{w}$ from data through Maximal Likelihood principle.

I'm looking for similar distribution but extended with "variance" parameter vector. A parameter that would represent the confidence in $\vec{w}$. Then I would like to infer both $\vec{w}$ and its confidence.

Does anybody of you happen to know such distribution or a research on the topic?

## 3 Answers

This question appears to confuse two distinct things. Any additional parameter in the model would (by definition) describe the distribution of $i$, not the distributions of any of the $w_i$. Unless you adopt a Bayesian prior for $\vec{w}$ (which does not seem to be part of this question), the parameters do not have any distribution at all: they are what they are. When you use a particular procedure to estimate $\vec{w}$, however, then the estimates $(\hat{w_i})$ do have a distribution. It makes sense to talk about the variance of that distribution. It can be estimated in standard ways, such as the inverse of the expectation of the Hessian of the log likelihood. It is unnecessary--and meaningless--to introduce yet another parameter to capture that information.

• (+1) good and a very fundamental point. – suncoolsu Jan 10 '11 at 2:47
• I thought, may be @Lew is trying to estimate the parameter and the corresponding CI. – suncoolsu Jan 10 '11 at 2:49

I am not sure, but I think your probability model is a special case of Multinomial logit model with no covariates and only the intercept terms ($w_i$ will be the intercepts).

This is model is a special case of GLM and hence there exits an iteratively weighted least square method (IRWLS) to get the maximum likelihood estimates of $w_i$.

If you don't want to code the IRWLS algorithm yourself, please check the polr function in the MASS library in R to accomplish the ML estimation of $w_i$.

EDIT

As @whuber points out, unless you adopt a Bayesian approach, there is no distribution for the parameters (assumed fixed), but there exits one for the estimates (here: ML) as they are estimated from the data, hence the randomness.

HTH

S.

The Maxwell distribution is the classical limit under conditions of high temperature and non-interacting wave functions of both Fermi-Dirac statistics and Bose-Einstein statistics. I would expect that you would want to look at the F-D statistics if you are interested in higher variance, since Bose-Einstein statistics lead to aggregations of particles, whereas F-D statistics (and the Pauli exclusion principle) are what keep neutron stars from further collapse:

(F-D without normalization): \begin{align} P(i| w_1, w_2, \ldots, w_n) = \frac{exp(w_i)}{\sum_{i=1}^n exp(w_i) +1} \end{align}

Edit: The Bradley-Terry model is a special case of a more general paired-choice model that was proposed by Stern:
In The Springer Encyclopedia of Mathematics you read: "H. Stern has considered, [a6], models for paired comparison experiments based on comparison of gamma random variables. Different values of the shape parameter yield different models, including the Bradley–Terry model and the Thurstone model. Likelihood methods can be used to estimate the parameters of the models. The likelihood equations must be solved with iterative methods."

H. Stern, "A continuum of paired comparison models" Biometrika , 77 (1990) pp. 265–273

http://biomet.oxfordjournals.org/content/77/2/265.abstract

• Well, I'm no expert in physics. I googled few "Fermi-Dirac statistics" articles, but I can't see any biparametric distribution. Also please note that I am not interested with higher variance but in variance as a parameter. – Łukasz Lew Jan 9 '11 at 23:23
• Comparing Gamma random variables is exactly the kind of idea I needed. How can I download the article? – Łukasz Lew Jan 11 '11 at 14:51
• Added link. Wasn't able to find source of it that was free but perhaps you can get it through your university library. – DWin Jan 11 '11 at 20:45