# Linear Regression with heavy tailed noise

The model is linear $y_i = a\cdot x_i + b + e_i,~ i = 1,2,\ldots,N$. It is given that the noise is heavy tailed. However the distribution of noise conditional on $x$ is the same for all data points. My question is that how should I model the data generating process? Should I use a Student-t distribution for the noise process? Should I use M estimator in R? Facts: 1. OLS is not to be used. 2. 2. The noise distribution can depend on $x$, but is independent across samples. 3. conditioned on the value of $x$, noise's mean is 0. More clarification: the distribution of noise at $x$, i.e., $N(x)$ is a function of $x$. However for different data samples $i=1,2,\ldots,M$, the distribution of noise samples remains the same.

• Is this an empirical problem (i.e. fitted model with assumptions being violated) or a theoretical (math) problem? – Jon Sep 21 '16 at 23:44
• I am given a data set containing $x_i,y_i$ and I have to estimate $a,b$ – user131929 Sep 22 '16 at 0:50
• – kjetil b halvorsen Feb 24 '17 at 8:20

Fat tailed $e$ does not violate Gauss--Markov. Without more context on your problem or assuming Gauss--Markov doesn't hold, good old OLS is the best linear unbiased estimator.

Are there any other aspects to the problem that you didn't mention?

• Following are the problem details: 1. Gaussian linear models are not to be considered because noise can be heavytailed. 2. The noise distribution can depend on $x$, but is independent across samples. 3. conditioned on the value of $x$, noise's mean is $0$. – user131929 Sep 22 '16 at 4:32
• dmsul -- If the distribution is sufficiently far from normal, all linear estimators may be bad. Beating a collection of other bad estimators is a pretty dubious crown. – Glen_b Sep 22 '16 at 4:41
• @user131929 1. Is this for a class? 2. You should include relevant information in the question rather in comments where people may miss it. 3 Your point 2 above contradicts the third sentence of your question. Please clarify the question to remove any ambiguity. 4 Heavy tails in the error distribution can (fairly dramatically) increase the problem with unusual points in x-space. What is known about the arrangement of x's? 5 There are various things one might try to do well at with a problem like this -- what are you trying to achieve, exactly? – Glen_b Sep 22 '16 at 4:47
• I have included the comments in the question itself. Regarding your comment, "Your point 2 above contradicts the third sentence of your question." I can't see any contradiction. Point 2 simply implies that the distribution of noise at $x$, i.e., $N(x)$ is a function of $x$. However for different data samples $i=1,2,\ldots,M$, the distribution of noise samples remains the same. – user131929 Sep 23 '16 at 6:58

You could use iterative feasible generalized least squares.

Start by setting weights for each datapoint to 1, i.e. no weighting, and use the following algorithm:

1. Fit a weighted regression model for each dataset using weights.
2. Create a single dataset combining squared residuals/errors, $e_i^2$ and their respective $x$ values.
3. Fit $e_i^2 = a\cdot x_i + b$. If the noise is zero mean, $e_i^2$ is equal to the variance of the error at $x_i$.
4. Update your weights with the squared errors prediction model
5. Go back to 1 until convergence.
• For the improvement of the community, please describe negative votes. – rafaelvalle Feb 24 '17 at 17:03