10
$\begingroup$

Say I have a $n$ pairs $(X_i, Y_i)$ and I have the relationship $$Y_i = \beta_0 + \beta_1X_i +\epsilon_i$$ where the error terms $\epsilon_i$ are iid from a student t-distribution with constant (known) degrees of freedom, (say $k\in\mathbb{N}$ degrees of freedom, where $k > 2$). If I wanted to compute the the MLE for $\beta_0, \beta_1$, how do I relate the given distribution to the parameters of interest?

Sorry if this is trivial, but I searched around and haven't really seen an example like this.

$\endgroup$
1

2 Answers 2

12
$\begingroup$

The MLE is obtained by maximising the log-likelihood function, so the first thing you will want to do is have a look at this function. Using the density function for the Student T-distribution with known degrees-of-freedom $k \in \mathbb{N}$, you can write the log-likelihood as:

$$\ell_\mathbf{x,y} (\beta_0, \beta_1) = - \sum_{i=1}^n \ln \Big( k + (y_i - \beta_0 - \beta_1 x_i)^2 \Big).$$

The residuals $r_i = y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i$ under the MLE minimise $\sum_{i=1}^n \ln ( k + r_i^2 )$. As $k \rightarrow \infty$ we have $\ln ( k + r_i^2 ) = \ln(k) + \ln(1+r_i^2/k) \approx \ln(k) + r_i^2/k$ so that the residuals minimise $\sum_{i=1}^n r_i^2$ in the limit, which is the standard OLS solutions for normally distributed errors. Use of the T-distribution effectively dampens the effect of large residuals, through the above logarithmic transformation, and so the MLE is more tolerant of having some large residuals than in the normal case.

Finding the MLE: The MLE can be obtained via numerical maximisation of the log-likelihood using ordinary calculus techniques. The gradient of the log-likelihood is given by the partial derivatives:

$$\begin{equation} \begin{aligned} \frac{\partial \ell_\mathbf{x,y}}{\partial \beta_0}(\beta_0, \beta_1) &= \sum_{i=1}^n \frac{2 (y_i - \beta_0 - \beta_1 x_i)}{k+(y_i - \beta_0 - \beta_1 x_i)^2}, \\[6pt] \frac{\partial \ell_\mathbf{x,y}}{\partial \beta_1}(\beta_0, \beta_1) &= \sum_{i=1}^n \frac{2 x_i (y_i - \beta_0 - \beta_1 x_i)}{k+(y_i - \beta_0 - \beta_1 x_i)^2}. \\[6pt] \end{aligned} \end{equation}$$

This leads to the score equations:

$$\begin{equation} \begin{aligned} 0 &= \sum_{i=1}^n \frac{2 (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)}{k+(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2}, \\[6pt] 0 &= \sum_{i=1}^n \frac{2 x_i (y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)}{k+(y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i)^2}. \\[6pt] \end{aligned} \end{equation}$$

These equations can be solved numerically using iterative techniques such as Newton-Raphson or gradient-descent methods, or other more complicated methods. The exact distribution of the MLE will be complicated, but for large samples the MLE should be normally distributed, according to standard large-sample theory.

$\endgroup$
4
  • 1
    $\begingroup$ Nice answer (+1), but you don't address some practical issues, such as potential non-uniqueness of solutions (there can be multiple local maxima in some situations; your score function approach should find a local maximum, and if you start it somewhere else it may find another; how should we make sure we at least find a fairly good one?). Here's a log-likelihood for an intercept-only model with $\nu>2$ on a small set of data i.sstatic.net/ffj5b.png $\endgroup$
    – Glen_b
    Commented Jun 10, 2018 at 2:19
  • 2
    $\begingroup$ Agreed (+1) - I guess this is just a teaser, since a longer exposition of calculus and iterative numerical methods would be a much bigger answer. (I don't really have a patience to give a longer answer!) The best practical thing for an OP who is not solid on this optimisation would be to plot the log-likelihood as a surface plot over the two dimensions of the parameters over the range near some identified critical points; that way he can see if these are local or global, without having to be very sophisticated with theory. $\endgroup$
    – Ben
    Commented Jun 10, 2018 at 3:00
  • $\begingroup$ Small correction, I think that as k goes to infinity we have that $ln(k+r_i^2)$ goes to $ln(k) + r_i^2/k$ $\endgroup$ Commented Apr 8, 2021 at 19:01
  • $\begingroup$ @DrorSpeiser: Thanks --- edited. $\endgroup$
    – Ben
    Commented Apr 8, 2021 at 23:20
1
$\begingroup$

It seems me that, in comparison to previous answer, easier result are available.

Infact if the $v$ parameter are known, the ML estimators for $\beta_0$ and $\beta_1$ mantain the usual OLS formulation. In other terms, the $\beta$s ML estimators under Student-t distribution for error terms are the same as in Normal case. However their variance are considerably inflated if $v$ is small. Moreover usual t and F test remain valid.

See, Arnold Zellner: Bayesian and Non-Bayesian Analysis of the Regression Model with Multivariate Student-t error terms - Journal of the American Statistical Association, Vol. 71, No. 354 (Jun., 1976), pp.400-405.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.