22
$\begingroup$

What are the maximum-likelihood estimators for the parameters of Student's t-distribution? Do they exist in closed form? A quick Google search didn't give me any results.

Today I am interested in the univariate case, but probably I will have to extend the model to multiple dimensions.

EDIT: I am actually mostly interested in the location and scale parameters. For now I can assume that the degrees of freedom parameter is fixed, and possibly use some numeric scheme to find the optimal value later.

$\endgroup$
  • $\begingroup$ To my knowledge they do not exist in closed form. A gradient ascent type approach may be required. $\endgroup$ – Pat Jul 8 '13 at 16:30
  • $\begingroup$ Although the Student t distribution has a single parameter, you refer to "parameters" in the plural. Are you perhaps including location and/or scale parameters? $\endgroup$ – whuber Jul 8 '13 at 19:42
  • $\begingroup$ @whuber, thanks for the comment, I am indeed interested in location and scale parameters, more than in the degrees of freedom. $\endgroup$ – Grzenio Jul 9 '13 at 9:51
  • $\begingroup$ With $n$ data, the likelihood equation for the location parameter is algebraically equivalent to a polynomial of degree $2n-1$. Do you consider a zero of such a polynomial to be given in "closed form"? $\endgroup$ – whuber Jul 9 '13 at 13:23
  • $\begingroup$ @whuber, are there any special cases for small n, e.g. n=3? $\endgroup$ – Grzenio Jul 10 '13 at 9:29
26
$\begingroup$

Closed form does not exist for T, but a very intuitive and stable approach is via the EM algorithm. Now because student is a scale mixture of normals, you can write your model as

$$y_i=\mu+e_i$$

where $e_i|\sigma,w_i \sim N(0,\sigma^2w_i^{-1})$ and $w_i\sim Ga(\frac{\nu}{2}, \frac{\nu}{2})$. This means that conditionally on $w_i$ the mle are just the weighted mean and standard deviation. This is the "M"step

$$\hat{\mu}=\frac{\sum_iw_iy_i}{ \sum_iw_i}$$ $$\hat{\sigma}^2= \frac{\sum_iw_i(y_i-\hat{\mu})^2}{n}$$

Now the "E" step replaces $w_i$ with its expectation given all the data. This is given as:

$$\hat{w}_i=\frac{(\nu+1) \sigma^2 }{\nu \sigma^2 +(y_i-\mu)^2}$$

so you simply iterate the above two steps, replacing the "right hand side" of each equation with the current parameter estimates.

This very easily shows the robustness properties of the t distribution as observations with large residuals receive less weight in the calculation for the location $\mu$, and bounded influence in the calculation of $\sigma^2$. By "bounded influence" I mean that the contribution to the estimate for $\sigma^2$ from the ith observation cannot exceed a given threshold (this is $(\nu+1)\sigma^2_{old}$ in the EM algorithm). Also $\nu$ is a "robustness"parameter in that increasing (decreasing) $\nu$ will result in more (less) uniform weights and hence more (less) sensitivity to outliers.

One thing to note is that the log likelihood function may have more than one stationary point, so the EM algorithm may converge to a local mode instead of a global mode. The local modes are likely to be found when the location parameter is started too close to an outlier. So starting at the median is a good way to avoid this.

$\endgroup$
  • 1
    $\begingroup$ That's awesome. I've been toying with the idea of fitting student t's using EM for a while for precisely the reason that it looks like a mixture of gaussians. Do you have a citation/reference for the update equations you give? Having that would increase the awesomeness of this post even further. $\endgroup$ – Pat Jul 10 '13 at 8:37
  • $\begingroup$ Actually, i think I've found one myself, for a mixture model of student t's (which I'm so going to use for stuff): The mixtures of Student’s t-distributions as a robust framework for rigid registration. Demetrios Gerogiannis, Christophoros Nikou, Aristidis Likas. Image and Vision Computing 27 (2009) 1285–1294. $\endgroup$ – Pat Jul 10 '13 at 8:46
  • $\begingroup$ The link in my answer to this question has a very general EM framework for loads and loads of likelihood functions - quantile, student, logistic, and does general regression. You specific case is "regression" without covariates - intercept only - so fits nicely into this framework. Also, there are a vast number of penalty terms that you can incorporate into this framework. $\endgroup$ – probabilityislogic Jul 12 '13 at 2:01
  • $\begingroup$ @probabilityislogic really neat! And what if $\nu$ is also unknown? Can you also give some reference please? Maybe best here: stats.stackexchange.com/questions/87405/… $\endgroup$ – Quartz Feb 21 '14 at 12:17
  • $\begingroup$ I think this reference is better than @Pat's. 'ML ESTIMATION OF THE t DISTRIBUTION USING EM AND ITS EXTENSIONS, ECM AND ECME.' You have to be very careful on the selection of initial parameter value while running the EM algorithm because of the local-optimum issue. In other words, you have to know something about your data. Usually, I avoid the use of the t distribution in my research. $\endgroup$ – user46849 Jun 6 '14 at 9:28
4
$\begingroup$

The following paper addresses exactly the problem you posted.

Liu C. and Rubin D.B. 1995. "ML estimation of the t distribution using EM and its extensions, ECM and ECME." Statistica Sinica 5:19–39.

It provides a general multivariate t-distribution parameter estimation, with or without the knowledge of the degree of freedom. The procedure can be found in Section 4, and it is very similar to probabilityislogic's for 1-dimension.

$\endgroup$
  • 7
    $\begingroup$ It sounds like the paper you refer to contains a useful answer to the question, but answers are better when they are standalone and do not necessitate outside resources (here, for example, it is possible that OP or readers do not have access to this paper). Could you flesh out your answer a little bit to make it more standalone? $\endgroup$ – Patrick Coulombe Nov 25 '15 at 23:47
3
$\begingroup$

I doubt that it exists in closed form: if you write any one of the factors of the likelihood as $$\frac{\Gamma(\frac{\nu+1}{2})} {\sqrt{\nu\pi}\,\Gamma(\frac{\nu}{2})} \left(1+\frac{t^2}{\nu} \right)^{-\frac{\nu+1}{2}} = \frac{\Gamma(\frac{\nu+1}{2})} {\sqrt{\nu\pi}\,\Gamma(\frac{\nu}{2})} \exp \left \{ \left [ \ln \left(1+\frac{t^2}{\nu} \right) \right ] \left [ {-\frac{\nu+1}{2}} \right ]\right \}$$ and take the ln of that, you will get a nonlinear equation in $\nu$. Even if you manage to get a solution, then depending on the number of factors (terms) $n$, the MLE equation is going to depend on this $n$ in a nontrivial way. All that dramatically simplifies, of course, when $\nu \rightarrow \infty$, when the power approaches an exponential (Gaussian PDF).

$\endgroup$
  • 1
    $\begingroup$ Even in the Gaussian setting the log likelihood is nonlinear in its parameters :-). $\endgroup$ – whuber Jul 8 '13 at 20:05
  • $\begingroup$ I am actually interested in location and scale parameters, more than in the degrees of freedom. Please see edit to the question, and sorry for being not precise. $\endgroup$ – Grzenio Jul 9 '13 at 9:54
2
$\begingroup$

I have recently discovered a closed-form estimator for the scale of the Student's t distribution. To the best of my knowledge, this is a new contribution, but I would welcome comments suggesting any related results. The paper describes the method in the context of a family of "coupled exponential" distributions. The Student's t is referred to as the Coupled Gaussian, where the coupling term is the reciprocal of the degree of freedom. The closed-form statistic is the geometric mean of the samples. Assuming a value of the coupling or degree of freedom, an estimate of the scale is determined by multiplying the geometric mean of the samples by a function involving the coupling and a harmonic number.

https://arxiv.org/abs/1804.03989 Use of the geometric mean as a statistic for the scale of the coupled Gaussian distributions, Kenric P. Nelson, Mark A. Kon, Sabir R. Umarov

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.