6
$\begingroup$

Sometimes I see advice to fit regressions with student-t residuals rather than using OLS (which is equivalent to assuming normally distributed residuals) if the distribution of the residuals is heavy-tailed. However, since the OLS estimator is BLUE (by Gauss-Markov), it should have lower variance (and therefore MSE) than a regression that assumes student-t residuals fit via maximum likelihood. This is true even if the residuals truly are t-distributed.

In a simple simulation (see bottom for R code), OLS and t-regression are essentially equivalent with respect to out-of-sample MSE and both achieve correct coverage for CIs for the $\beta$ coefficient even though the true residuals follow a student-t distribution. If OLS performs equally for standard tasks such as prediction (as judged by MSE) or inference (as judged by coverage of CIs for model parameters), what are the advantages of fitting a regression assuming student-t errors? Shouldn't I benefit by estimating parameters assuming a correctly specified likelihood rather than a misspecified likelihood? Or is my simulation misleading?

An answer to this post suggests that prediction intervals will be wrong if one fails to use t-distributed errors. Is that really the only benefit?

Here's the simulation code:

library(hett) # for fitting t-based regressions

get_regression_function <- function(beta) {
  reg_function <- function(x){
    beta[1] + beta[2] * x
  }
  return(reg_function)
}
get_t_prediction <- function(df, model) {
  t_beta_hat <- model$loc.fit$coefficients
  design_mat <- df$x
  intercept <- rep(1, length(df$x))
  design_mat <- cbind(intercept, design_mat)                 
  pred <- design_mat %*% as.matrix(t_beta_hat)
  return(pred)
}

get_ols_ci_coverage <- function(ols_fit, par_name, true_theta) {
  CI_mat <- confint(ols_fit)
  lw <- CI_mat[par_name, '2.5 %']
  up <- CI_mat[par_name, '97.5 %']
  cover_beta_1_ols <- (lw <= true_theta) & (true_theta <= up)
  return(cover_beta_1_ols)
}

get_t_ci_coverage <- function(t_fit, par_name, true_theta) {
  t_summary <- as.data.frame(summary(t_fit)$loc.summary$coefficients)
  t_pt_est <- t_summary[par_name, 'Estimate']
  t_se <- t_summary[par_name, 'Std. Error']
  t_ci <- t_pt_est + 1.96 * t_se * c(-1, 1)
  cover_t <- (t_ci[1] <= true_theta) &
    (true_theta <= t_ci[2])
  return(cover_t)
}

## simulation
true_beta_0 = 0.5
true_beta_1 = 1.5
reg_function <- get_regression_function(
  beta = c(true_beta_0, true_beta_1))

ssr_ols <- c()
ssr_t <- c()
ols_ci_coverage <- c()
t_ci_coverage <- c()

num_sims <- 10000
set.seed(1)
for (i in 1:num_sims){
  # Generate a dataset of size N
  N = 10000
  x <- rnorm(n = N, mean = 0, sd = 1)
  errors <- rt(N, df = 3)
  y_mean <- reg_function(x = x)
  y <- y_mean + errors
  df <- data.frame(y = y, x = x)
  df_train <- df[1:(N/2), ]
  df_test <- df[(N/2 + 1):N, ]
  
  # fit linear models
  OLS_fit <- lm(y ~ x, data = df_train)
  t_fit <- tlm(y ~ x,
               data = df_train)
  
  # generate test set predictions
  lm_pred <- predict(OLS_fit, newdata = df_test)
  t_pred <- get_t_prediction(
    df = df_test, model = t_fit)
  
  # test residuals
  lm_res <- lm_pred - df_test$y
  t_res <- t_pred - df_test$y
  
  # MSE -- test set
  ssr_ols[i] <- mean(lm_res^2)
  ssr_t[i] <- mean(t_res^2)
  
  # CI coverage for beta1
  ols_ci_coverage[i] <- get_ols_ci_coverage(
    ols_fit = OLS_fit, par_name = 'x', true_theta = true_beta_1)
  t_ci_coverage[i] <- get_t_ci_coverage(
    t_fit = t_fit, par_name = 'x', true_theta = true_beta_1)
}

mean(ssr_ols)
mean(ssr_t)
mean(ols_ci_coverage)
mean(t_ci_coverage)
$\endgroup$
3
  • $\begingroup$ Se the almost dup: stats.stackexchange.com/questions/120776/… $\endgroup$ Jan 19 at 0:23
  • $\begingroup$ @kjetil Might be worth closing on that basis $\endgroup$
    – Glen_b
    Jan 19 at 0:36
  • 1
    $\begingroup$ You don't mean "residuals:" you mean errors. A residual is what you observe relative to a model that you fit; the errors describe the conditional distribution of the response in your model. $\endgroup$
    – whuber
    Jan 19 at 16:22

4 Answers 4

12
$\begingroup$

Here's one reason:

If you fit the parameters using maximum likelihood, with an assumption of t-distributed errors the fitted line is less impacted by points that are far away from the bulk of the data (in the y-direction).

If you have heavy tailed errors, it's easy to get points that pull the line up (away from the bulk of the data) if you fit by least squares. Using a heavy-tailed error distribution down-weights those points so they don't pull the line about so much.

However, it's important to note that this robustness to heavy tails in the y-values doesn't offer protection against observations with high influence. If that's likely to be an issue, you need something that's robust against influential outliers. [With a designed experiment where you control the predictor-values this may not be an issue, but you don't always have designed experiments.]

(There are other reasons you might want to use a suitable model for errors, like more efficient estimates of parameters and the possibility of more suitable small-sample inference.)


The point about least-squares being BLUE is a bit misleading. Yes, it is best among linear unbiased estimators, but if your conditional distribution is very heavy tailed all linear estimators may be very poor, even arbitrarily so, and having the best of a very poor collection of estimators is no consolation. If you want a reasonable level of efficiency in the presence of heavy tails, linear estimators are not a good choice.

$\endgroup$
4
  • 1
    $\begingroup$ Thanks! That makes intuitive sense, but how does it translate to poor model performance more concretely? Is it possible to show that predictions are worse in some sense (e.g. MSE) or that inference on the parameters is harder? Do problems only emerge in small samples or will large datasets also exhibit problems? I'm trying to understand why my simulation example doesn't show an obvious benefit to use t regression. Thanks again. $\endgroup$
    – frelk
    Jan 19 at 1:12
  • $\begingroup$ I am not a statistician, but doesn't this answer just kick the can down the road? Sure, we may decide that a heavy-tailed residual distribution is more appropriate for a given dataset, but among the various heavy-tailed distributions out there, why favor a T distribution? $\endgroup$
    – Max
    Jan 20 at 7:12
  • $\begingroup$ I don't think that's within the scope of the question that was asked (I think that's a new question - if that had been asked I'd have given a very different answer). In many cases personally I'd choose other heavy-tailed distributions (I would NOT typically favour a t-distribution but that WASN'T the question being asked). Nonetheless, in some specific cases the t makes for a good approximation. Outside that, it covers a wide variety of tail behavior from slightly heavier-than-normal to cases where even low order population moments are not finite, so it can offer a flexible range of symmetric $\endgroup$
    – Glen_b
    Jan 20 at 7:19
  • $\begingroup$ ... heavy-tailed distributions. Most of the points I made are not specific to the t-distribution but apply to a wide variety of heavy-tailed choices, but that wasn't what was asked by the OP. I think it's fine for the OP to raise issues that they asked that they feel I didn't adequately address but the answer isn't to blame for failing to address things that were not asked. $\endgroup$
    – Glen_b
    Jan 20 at 7:20
7
$\begingroup$

However, since the OLS estimator is BLUE (by Gauss-Markov), it should have lower variance (and therefore MSE) than a regression that assumes student-t residuals fit via maximum likelihood.

The estimator that assumes student-t residuals is not a linear estimator. OLS is the Best among Linear Unbiased Estimators. So, that does not include the student-t residuals estimator. The variance of the estimator that uses student-t residuals can be smaller than the variance of the OLS estimator.

The student-t estimator has the advantage that it is more robust to violations of the assumption that the error is Gaussian distributed. In cases of relatively more higher values it can have a lower variance than the OLS estimator.

An extreme and simple example is the estimation of the location parameter of a shifted t-distribution with 1 degree of freedom (which is the same as the Cauchy distribution). In that case the OLS estimator is the mean of the sample and will have undefined variance. On the other hand, the MLE estimate of the Cauchy distribution although difficult to compute, has lower variance

An answer to this post suggests that prediction intervals will be wrong if one fails to use t-distributed errors. Is that really the only benefit?

The linear estimator, a weighted sum of all the $y$, has the distribution of a sum of the distributions of the $y$. Except in extreme cases, this will approach a normal distribution quickly and it wouldn't be so wrong to for prediction intervals.

$\endgroup$
2
$\begingroup$

The weakness of the OLS regression is that it is extremely sensitive to outliers, (the influence of the outlying points on the loss function grows as the square of their distance from the regression line). One of the possible alternatives for making regression more robust (i.e., less sensitive to the outliers, see Robust regression) is using loss functions that give less weight to the outlying points, such as Huber loss or Tukey loss (see also M-estimator).

Student-t distribution with its long tails is also a potential candidate with distinct advantages and disadvantages:

  • It is more difficult to work with analytically than Huber or Tukey loss functions
  • It is a real probability distribution (unlike Huber and Tukey, which are somewhat ad-hoc choices), and smoothly fits, e.g., with Bayesian methods.
$\endgroup$
-2
$\begingroup$

Student-t Distribution is the distribution of small sample; as opposed: one of the important assumptions of using OLS - is Normal Distribution, that needs great amount of independent variables in the sample... and in order to cope with outliers - you'd better to increase the size of your sample-data to reach its normality (though, of course, you can just remove your outliers from your data analysis - the other way to cope with robust vars)... to check the data for normality you can use Shapiro-Wilk normality test or Kolmogorov–Smirnov test...

Anyway, you cannot compare OLS-regression & testing residuals - because in order to trust R^2 score of your regression curve (you're getting with OLS instrument) - you have to check your residuals for Normality as well after OLS...

So, OLS is having rather strict requirements about the structure & size of the sample - Normality (of data & residuals as well)... but if you cannot prove the normality you'd better use MLE (maximum likelyhood estimation) instead of OLS - to find out more appropriate distribution to your emperical data... Also speaking about OLS - it also requires the f(x) to be unambiguously defined in apriory analysis of your data dependences, otherwise MLE is advised to find the theoretical curve that mostly suits your emperical distribution...

But to prove that your regression coeficients for each X are meaningful - yes, you see t-Student criteria for each of coef's of regression model found, - additionaly to the F-criteria for the whole regression model found...

import statsmodels.api as sm
...
model= sm.OLS(y,x)
results= model.fit()
print(results.summary())

be carefull with terms: t-Student tests can be used in different stat.investigations with different sense & t-Student Distribution can describe a "small sample"...

$\endgroup$
3
  • $\begingroup$ This answer seems to reflect an unusual interpretation of the question. The issue is not one of small samples, but rather a model in which errors (mistakenly called "residuals" in the question) are assumed to follow some particular distributional family; namely, Student t distributions. Confirming this is the line errors <- rt(N, df = 3) (found in the middle of the code) in which the errors are explicitly drawn from a Student $t(3)$ distribution. The dataset is of size $N=10^4:$ that's not a small one! $\endgroup$
    – whuber
    Jan 19 at 16:21
  • $\begingroup$ the question itself seems to mix basic fundamentals of regression analysis - therefore I clarified terminology, because the author was asking WHY & we do not know in what a context "benefit of regression with student-t residuals over OLS regression?" - MY ANSWER IS - no benefit - becase the distribution of data & distr of residuals are different data!.. & concerning the term "student-t residual" I'm almost sure - it was just about student-t scores of coefs in regression equation... of course 10^4 is not a small one, but "student-t residuals" is not about the sampling- I advise NOT MIX TERMS $\endgroup$
    – JeeyCi
    Jan 20 at 3:40
  • $\begingroup$ well, really - studentization is a way "to return residuals back to the same level of conditional variance as the unobserved model errors ε are" - as I noticed here - stats.stackexchange.com/questions/306735/…... well, I just remembered that "the fact that you have a large studentized residual does not necessarily mean that the observation is an outlier" BUT NotaBene: stats.stackexchange.com/questions/406407/… $\endgroup$
    – JeeyCi
    Jan 21 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.