5
$\begingroup$

I am running a regression and find coefficients of my explanatory variables very interesting (they are dummy variables). When I informed that to my lecturer, he told me that there are not much variations in the explanatory variables then the results might not very good. (I did not check that before because I jumped to the regression without descriptive statistics, I have learned my lesson)

I feel vaguely that if the explanatory variable does not vary much, then it might not be a good regressor. But anyone can provide some proof/intuition/reading on the issue? Can we get around that problem?

For the explanatory variable in my example, 94% of the sample are native (1) and 6% are foreign. I don't think there is a sample selection bias here because the data are collected in a country so why would one expect to have a balanced nationality.

$\endgroup$

1 Answer 1

5
$\begingroup$

The basic idea is that more variation in the regressors allows you to more confidently pin down the relationship between $y$ and $X$ in your regression.

Recall that the slope coefficient of the estimated regression is to provide an answer to the question: "How does $y$ change, on average, when $X$ changes by one unit?" Now, in the limit, if $X$ does not vary at all, the sample is of course entirely uninformative about that question. If $X$ only varies a little, the slope will be estimated very imprecisely.

For example, when you want to estimate the returns to more education in terms of additional salary, but your sample consists of, say, $n-1$ students with a BSc degree and one with an MSc, the return to completing an MSc will be estimated from that student alone. Since that student may have all sorts of pecularities, the effect on salary from completing an MSc will be estimated very poorly.

Here is a simulated example where the purple dots are pairs $(x_i,y_i)$ where the regressors are uniform on $[0,1]$, whereas the golden dots are from regressors uniform on $[-2,3]$, thus their variation is larger.

The true $\beta$ is .5 in both cases, and the errors are $N(0,0.5^2)$ (the code is below for completeness), $n=50$.

As we observe $X$ over a wider range in the golden case, regression finds it easier to spot the slope of the true regression line. We see that the estimated slope for the purple sample is quite off (in fact, even negative), whereas the golden one is quite good. Of course, the figure is for a single sample and thus only suggestive. You could run the code many times (aka, a simulation study) to assess the difference in precision for this (or of course other) design(s).

enter image description here

One can also show this effect analytically: In a regression with constant (as in my example), the variance of the estimated slope coefficient is (with $\sigma^2$ the variance of the error terms)

$$ Var(\hat{\beta})=\frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2} $$

We notice that as the variation in $X$ increases, the variance of the OLS estimate decreases.

Code for the plot:

n <- 50
u <- rnorm(n,sd=.5)
beta <- .5

X1 <- runif(n,0,1)
X2 <- runif(n,-2,3)

y1 <- X1*beta + u
y2 <- X2*beta + u

plot(X1,y1,xlim=c(-2,3),ylim=c(-2,3),col="purple",pch=19)
abline(a=0,b=beta)
abline(lm(y1~X1),col="purple")
points(X2,y2,col="gold",pch=19)
abline(lm(y2~X2),col="gold")
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.