# Effect of covariates distribution on linear regression

If the covariates $X_1,X_2$ are distributed as follows, what effect does it have on the linear model $y = \beta_0 + \beta_1 X_1 + \beta_2 X_2$

$X_1,X_2$ do not seem to exhibit a strong correlation, so collinearity can be ruled out. • think about outliers – Christoph Hanck Dec 11 '15 at 9:05
• It would be illuminating to remove the outliers, fit a regression, put them back, fit another. – Matthew Drury Dec 11 '15 at 18:49

Not to be overly pedantic, but those 'outliers' visible in the graph of $X_1, X_2$ scatterplot are not what we normally refer to as outliers in the context of regression. For regression, outliers are observations with large (absolute value) residuals. When you have combinations of explanatory variables ($X_1, X_2$) that fall outside the pattern of most observations, these are INFLUENTIAL points. They have an abnormally high influence on estimation of the slopes in your model (and on the predictions given by your model when you extrapolate).

These are also called high-leverage observations. For instance consider the data:

x1 = c(15, 15, 22, 17, 10, 15, 23, 9, 18, 19, 60, 15)
x2 = c(27, 21, 35, 16, 17, 20, 19, 30, 17, 27, 30, 80)
y = c(11.9, 15.7, 18.4, 9.6, 7.4, 11, 16.9, 12.8, 11, 12.7, 24.5,
22.5)


Here, there are extreme (explanatory) values at (60, 30) and (15, 80), so these are high influence observations. A slight change in the y value at these locations will have a big influence on the fitted value.

> summary(lm(y~x1+x2))

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  4.06440    1.60958   2.525 0.032494 *
x1           0.25994    0.05067   5.130 0.000619 ***
x2           0.18809    0.03868   4.863 0.000892 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.236 on 9 degrees of freedom
Multiple R-squared:  0.8491,    Adjusted R-squared:  0.8156
F-statistic: 25.32 on 2 and 9 DF,  p-value: 0.0002013

> y
 24.5
> y = 10
> summary(lm(y~x1+x2))

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  8.91261    2.23551   3.987  0.00317 **
x1          -0.03901    0.07037  -0.554  0.59283
x2           0.18358    0.05372   3.417  0.00766 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.105 on 9 degrees of freedom
Multiple R-squared:  0.5701,    Adjusted R-squared:  0.4746
F-statistic: 5.968 on 2 and 9 DF,  p-value: 0.02239

> y
 22.5
> y = 10
> summary(lm(y~x1+x2))

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 12.665278   2.577255   4.914 0.000831 ***
x1          -0.004558   0.081130  -0.056 0.956426
x2          -0.010320   0.061933  -0.167 0.871340
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.58 on 9 degrees of freedom
Multiple R-squared:  0.003453,  Adjusted R-squared:  -0.218
F-statistic: 0.01559 on 2 and 9 DF,  p-value: 0.9846


If I changed the y value at another location (low leverage) I'll get little change in the model:

 > summary(lm(y~x1+x2))

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 12.665278   2.577255   4.914 0.000831 ***
x1          -0.004558   0.081130  -0.056 0.956426
x2          -0.010320   0.061933  -0.167 0.871340
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.58 on 9 degrees of freedom
Multiple R-squared:  0.003453,  Adjusted R-squared:  -0.218
F-statistic: 0.01559 on 2 and 9 DF,  p-value: 0.9846

> y
 7.4
> y = -3
> summary(lm(y~x1+x2))

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  9.79553    4.22547   2.318   0.0456 *
x1           0.04734    0.13302   0.356   0.7301
x2           0.02415    0.10154   0.238   0.8173
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 5.87 on 9 degrees of freedom
Multiple R-squared:  0.0202,    Adjusted R-squared:  -0.1975
F-statistic: 0.09277 on 2 and 9 DF,  p-value: 0.9123


There are two outliers one on the top right and one on the bottom left. They are outliers considering $(X_1,X_2)$ together but not individually.

Since determining the coefficients requires minimizing the squared error, presence of outliers will affect the magnitude of the coefficients.