1
$\begingroup$

Assume that OLS regression of the form:

$$Y_t = X_t'\beta + u_t$$

Suppose $X_t$ are stochastic, thus standard Gauss-Markov assumptions need to be accommodated. Given that:

$$\text{E} {(\hat\beta)} = \beta + \text{E}((X'X)^{-1}X'u)$$

Now, for OLS to be unbiased we need to additionally assume no covariance between the error term and the Xs. (Even more correctly, that they are independent).

Problem: Isn't assuming that $X_t$ is random similar to assuming that there is measurement error in X (That is $X_t = X_t^* + h_t$), where $h_t$ is a random white noise process. Now, this type of random measurement error in fact guarantees that the error term and independent variables are correlated and the beta parameters are downwardly biased. So clearly these are distinct cases, what is the difference? I don't see how a random variable could not be correlated with the error. Would it be correct to assume that the assumptions break down and betas are biased downward in practice, when random variables are introduced?

$\endgroup$
  • 5
    $\begingroup$ Your commentary in English does not describe the mathematical model you have written (assuming the second "$u_t$" is not the same as the first "$u_t$"!): there is no mathematical implication of correlation. Moreover, a "random" distribution of $X_t$ is not the same as measurement error, by any means! For instance, in an observational study the $X_t$ values could be things like species, color, or other nominal qualities that can be known without any measurement error at all, yet--because the subjects are obtained randomly from a population--all the $X_t$ values are a fortiori random. $\endgroup$ – whuber Apr 8 '16 at 16:01
  • $\begingroup$ @Whuber In other words, are you saying that the randomness here does not actually imply that there is a random component or error to X or that X is even a random variable. Rather that it's obtained randomly without choosing its value. Very different definition of random variable than for example: upa.pdx.edu/IOA/newsom/mlrclass/ho_randfixd.pdf (page 3), which matches the equation in the question, or any other sort of random process. Ie. The same type of random variable as $y_t$ $\endgroup$ – Dole Apr 8 '16 at 17:56
  • $\begingroup$ I'm afraid I do not follow the distinctions you are attempting to make between "random component," "random variable," and "obtained randomly." We have some threads on the definition of random variables that might help you, such as stats.stackexchange.com/questions/50. $\endgroup$ – whuber Apr 8 '16 at 18:00
  • $\begingroup$ @whuber Think about $y_t$, that is what I mean by a random variable. In other words, there is dependency on X, but there is also a random component or if you will, a measurement error $u_t$ to $y_t$. It's not merely randomly obtained, but actually has a random distribution. I will read through the question and edit this one accordingly. $\endgroup$ – Dole Apr 8 '16 at 18:07
  • $\begingroup$ @Dole would a toy simulation be helpful? do you know the language R? $\endgroup$ – Adrian Apr 8 '16 at 19:47
3
$\begingroup$

You ask "isn't assuming that $X$ is random similar to assuming that there is measurement error in $X$", in the sense that it biases $\hat{\beta}$?

The answer is no. Here's a little R simulation you can play with to help convince yourself:

get_df <- function(n_obs=10^3, true_beta=c(5, -1, 10, 5)) {
    stopifnot(length(true_beta) == 4)  # Coefficients on constant, x1, x2, x3
    df <- data.frame(x1=rnorm(n_obs), x2=rnorm(n_obs), epsilon=rnorm(n_obs), constant=1)
    df$x3 <- 2*df$x1 + df$x2 + rnorm(n_obs)
        df$y <- as.matrix(df[, c("constant", "x1", "x2", "x3")], n_obs, 4) %*% true_beta + df$epsilon
        df$x1_noisy <- df$x1 + rnorm(n_obs, sd=5)
    return(df)
}

get_beta_hat <- function(df, formula=y ~ 1 + x1 + x2 + x3) {
    fit <- lm(formula, data=df)
    return(coefficients(fit))
}

set.seed(543299)

beta_hat <- t(replicate(1000, get_beta_hat(get_df())))
colMeans(beta_hat)  # Very close to true values of (5, -1, 10, 5), even with stochastic X
apply(beta_hat, MARGIN=2, FUN=sd)

beta_hat_noisy_x1 <- t(replicate(1000, get_beta_hat(get_df(), y ~ 1 + x1_noisy + x2 + x3)))
colMeans(beta_hat_noisy_x1)  # I got (5, -0.01, 10.4, 4.6)
apply(beta_hat, MARGIN=2, FUN=sd)

The code simulates data where $$Y \equiv X\,\beta + \epsilon,$$with $X$ and $\epsilon$ (and therefore $Y$) random. Since $$\mathbb{E}\left[\epsilon | X\right] = 0,$$we have $\mathbb{E}\left[\hat{\beta} | X\right] = \beta$, i.e. our estimates of $\beta$ are unbiased conditional on any realization of $X$. That implies that they are unbiased unconditionally: $$\mathbb{E}\left[\hat{\beta}\right] = \mathbb{E}\left[\mathbb{E}\left[\hat{\beta} | X\right]\right] = \beta.$$

The issue with measurement error -- the reason it's different from randomness in $X$ -- is that measurement error does not enter the definition of $Y$. In other words, if you regress $Y$ on noisy $X$, you are fitting the wrong model; the true model regresses $Y$ on non-noisy (but possibly random) $X$.

$\endgroup$
  • $\begingroup$ Stellar answer! Indeed I assumed that the randomness of X meant that it is a random process dependent on y and not that the x's are randomly selected/sampled. Is the random sampling property only required for the case where X is stochastic then? Or is this yet a other distinct property? $\endgroup$ – Dole Apr 9 '16 at 14:01
  • $\begingroup$ @Dole what is your definition of "the random sampling property"? I don't entirely follow your question. You do seem to say that $X$ and $Y$ are independent, which is certainly not the case (unless $\beta = 0$). $\endgroup$ – Adrian Apr 9 '16 at 15:01
  • $\begingroup$ I mean the Gauss Markov assumption of random sampling IE the values are randomly obtained from the population,. I suppose that assumption is only needed if X is stochastic. $\endgroup$ – Dole Apr 9 '16 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.