Is it possible to use generated non-normal errors with a linear regression model

Question

I am working on linear regression models including classic and robust linear regression models.

By classic models, I mean ordinary least square and least absolute regression. Also, by robust models, I mean something like Huber regression and MM estimate.

I am building models on generated data in R, and I assumed that the coefficients are set to be zero Without loss of generality. Thus, Y=epsilon.

Now, Is it possible to generate Y from different distributions rather than the normal distribution and the same for the X matrix? For example, from t, log normal, Beta, Weibull, Exponential Distribution.

In other words, Is it possible for example to generate data from t-distribution for both Y and X, and then compare OLS and Huber Regression? Does this contradict the assumption of the Linear Regression where the errors assumed to be normally distributed?

Bear in mind

• My idea is to check/investigate which model is more robust to the outliers.

• I want for example to compare Ordinary least square and Huber regression, so first, I will generate X and Y from a normal distribution and then I compare them based on some measure. I will then repeat the process BUT I generate the Y and X from another distribution for example Beta or Expnonetioal. In this case, is it ok to generate data from latter distributions to build Linear Regression Models?

• The model each time will build on Y, and X, for example in R; lm(Y~X,...).

Answer 1

I think you'll want to revisit how regression simulation in supposed to work. The idea is to code the Data Generating Process (DGP) and as a result you will know what is true in the population. Then you build a sample using this DGP and run a regression on this sample (can be linear regression or Huber or any other type). You'll want to see if you recover the population parameters and how your conclusion changes as you change the DGP as desired. This way you can answer any questions you might have (such as investigating robustness to outliers). To simulate the model $Y=\beta_0+\beta_1*X+\epsilon$, you do the following:

1. Independently simulate the error term $\epsilon$ from any assumed distribution (can be any distribution you please), and the independent variable from any assumed distribution (can be any distribution you please). You can build in a dependency of $\epsilon$ and $X$ but keep in mind that (if X is a random variable) your OLS model necessarily assumes that $E(\epsilon_i|X_i)=0$, so this condition must be satisfied by whatever dependency relationship you decide to model. With X non-random, there can be no dependency between $\epsilon$ and $X$, by definition.
2. Assume values for $\beta_0$ and $\beta_1$. It looks like you want to assume that $\beta_0=0$ and $\beta_1=0$. That's fine but then X does not even enter the picture here. Your model ignores X and is literally $Y_i$=$\epsilon_i$. There is loss of generality once you assume $\beta_1=0$. You can assume $\beta_1=4$ without loss of generality, but not $\beta_1=0$.
3. Generate $Y$ values using the DGP. You do not assume another distribution for $Y$. The distribution of $Y_i$ in your case is identical to the distribution of $\epsilon_i$. Whatever you assumed for $\epsilon_i$ holds for $Y_i$ because $Y_i$=$\epsilon_i$.

This clarification of how regression simulation works should answer all your questions.

Answer 2

Not sure what is the question here. First of all, yes, you can simulate data using any data generating process. However, if what you want is to compare the scenario to data simulated from a normal distribution, you just need to make sure that in both cases, the theoretical variance and means are the same. For example, if: $$y=X\beta +e$$ $$e\sim N(0,1)$$ then you can compare it to: $$ e \sim U \left( -\frac{\sqrt(12)}{2},\frac{\sqrt(12)}{2} \right) $$ because in both cases the mean is 0 and variance 1.

Answer 3

is it ok to generate data from latter distributions to build Linear Regression Models?

You can simulate whatever you want. What matters more perhaps is the type of conclusions you want to draw from the results: I can imagine people using linear regression when the true error distribution is $t$-distributed, and it is valuable to know how linear regression performs vs other techniques. But is anyone using ordinary linear regression for exponential, or Weibull distributed processes?

---

Since this has the r tag, perhaps what you are really asking for is how to draw samples from non-normal distributions in R.

This is actually really simple and there are a lot of built-in choices, including each choice you mentioned:

• rnorm(n, mu, sd) for n random draws from a normal distribution with mean mu and variance sd^2;
• rt(n, df) for n random draws from a $t$-distribution with df degrees of freedom. You could multiply these draws by some value to increase their variance if you wanted to compare them to the normally distributed ones;
• rlnorm(n, meanlog, sdlog) for n random draws from a log-normal distribution with mean and standard deviation on the log scake given by meanlog and sdlog, respectively;
• rbeta(n, shape1, shape2) to draw n samples from a beta distribution with $\alpha$ given by shape1 and $\beta$ by shape2;
• rweibull(n, shape, scale) for n samples from a Weibull distribution with $k$ given by shape and $\lambda$ by scale;
• rexp(n, rate) for n samples from an exponential distribution with $\lambda$ given by rate.

There are many other distributions available in base R, as well as from packages you can install. You could even write your own random number generator based on runif to draw samples from an arbitrary probability distribution. For example, summing two uniform distributions with the same $a$ and $b$ will give you a triangular distribution.

You may then also want to set a seed before making a call to these functions. This forces the pseudo-random number generator to start from a fixed point, so any time you run your script, you will end up with the same 'random' numbers. This can be done by placing set.seed(xxx) at the top of your script, where xxx is some arbitrarily chosen integer. This makes your results reproducible.

Answer 4

It is certainly possible to generate random variables from distributions other than the normal distribution in R. You can find a large list of probability distributions here. If you are interested in conducting simulation analysis looking at the effects of outliers, a natural thing to do would be to use a T-distribution for the error term, which allows you to vary the degrees-of-freedom parameter which affects the "fatness of the tails" of the distribution. This can be done by generating random variables using the rt function.

Here is an example where I simulate linear regression data using a normally distributed explanatory variable and error terms that are T-distributed with two degrees-of-freedom (giving infinite variance). This is an error distribution with "fat tails" so you get some large error terms in the data.

#Set regression parameters
n     <- 200
beta0 <- 24.2
beta1 <- 0.8
    
#Generate simulated linear regression data
#Error terms are T-distributed with two degrees-of-freedom
set.seed(81301856)
ERR   <- rt(n, df = 2)
X     <- rnorm(n, mean = 40, sd = 8)
Y     <- beta0 + beta1*X + ERR

#Show scatterplot of the data
plot(X, Y, ylim = c(0, 100), main = 'Scatterplot of simulated regression data')