What does it mean for the residuals to follow a Student's t distribution?
The meaning depends on the degrees of freedom, $v$, of the Student's-t Distribution. For background information the Wikipedia entries on the Cauchy, Student's-$t$, and normal distributions are useful reading. I have not seen anything theoretically quantify the regression errors for the Student's-$t$ distribution from ordinary least squares (OLS) regression. For maximum likelihood iff degrees of freedom > 2 one can use MatLab, also see ref. Cross-Validated. It is easy enough to show what the effects are using simulations.
For 1 degree of freedom, the Student's-t becomes a Cauchy distribution and OLS regression is not useful, because the mean of a Cauchy distribution is undefined (therefore unstable when calculated). See the first image, Monte Carlo simulation of 10000 Cauchy residuals for sequential equidistant x-axis points, below with regression equation $y=3.50054 x-19394.7$ where the truth is $y=x$
When the degrees of freedom are greater than 1 up to and including 2, the mean is defined but the standard deviation (second moment) is not defined, and then the OLS regression is more accurate but not exact, see next image with $v=2$, and regression equation $1.04074 x-337.02$.
As $v$ increases the regression accuracy does as well. At $v>3$, the third moment, skewness is defined. For $v>4$ kurtosis is defined, and so on for higher moments and degrees of freedom.
To get some idea of what this means in practice, I made sequential models for increasing $v$ on the $x$-axis using two sets of random numbers, which coincidentally approach a slope of one and intercept of zero from opposite directions. First the intercepts:
Next the slopes:
As $v\to \infty$ the Student's-$t$ distribution becomes a normal distribution. Thank-you for a very interesting question that I was wondering about myself.