5
$\begingroup$

I am trying to learn about bias in simple linear regression. Specifically, I want to see what happens when the $cov(e,x) = 0$ assumption of the simple regression is violated.

If this assumption is violated, I arrive at

\begin{equation} \hat{\beta}_1 \rightarrow \beta_1 + \frac{{cov(e,x)}}{{var(x)}}. \end{equation}

This derivation is from this web page (equations 1 through 6). The web page says

If cov(e,x) =\= 0, the OLS estimator is inconsistent, i.e. its value does not converge to the true value of the parameter with the sample size. Moreover, the OLS estimator is biased.

To me, it is clear that $\hat{\beta}_1$ converges to a value that is not the true value $\beta_1$, so that makes it biased. However, the web page seems to conclude that this makes it inconsistent. Somehow, they conclude that the estimator is biased, but I am not sure (they simply use "moreover").

So here are my questions:

  1. What is the difference between bias and inconsistency in this case? (When they conclude that it is inconsistent, I conclude that it is biased.)
  2. Does it ever make sense to say that $\beta_1$ is biased? Or, can only an estimator $\hat{\beta_1}$ be biased?
  3. If the $cov(e,x) = 0$ assumption is violated, how can I find out what happens to the variance of $\hat{\beta}_1$? Can I tell if it increases or decreases?

EDIT To clarify question 3, I am wondering if there is a proof/argument for:

When the second assumption ($cov(e,x) = 0$) of Ordinary Least Squares is violated, the variance of $\hat{\beta_1}$ changes.

The only answer I can think of is using the result from omitted-variable bias. That is, comparing $var(\hat{\beta_1})$ and $var(\tilde{\beta_1})$ using the equations

\begin{align} var(\hat{\beta_1}) = \sigma^2/[SST_1(1-R_1^2)] \end{align}

\begin{align} var(\tilde{\beta_1}) = \sigma^2/SST_1. \end{align}

The full argument for the omitted-variable case comes from Wooldridge's text.

Since having an omitted variable is sufficient to violate $cov(e,x) = 0$, is the argument given by Wooldridge sufficient to prove that the variance is less than it would be if $cov(e,x) = 0$ held true?

(If my understanding is correct, I think that $\tilde{\beta_1}$ is the assumption-violating case. $\hat{\beta_1}$ is the 'true' case.)

$\endgroup$
3
  • $\begingroup$ What is the source for your new quote: When the second assumption ($cov(e,x) = 0$) of Ordinary Least Squares is violated, the variance of $\hat{\beta_1}$ changes. $\endgroup$
    – cwackers
    Nov 18, 2015 at 0:00
  • 1
    $\begingroup$ This is my supposition. It may very well be incorrect; there is no change. I would just like to prove it one way or the other $\endgroup$
    – Wise Owl
    Nov 18, 2015 at 13:14
  • $\begingroup$ Thanks. In your post above, to avoid confusion, it would help if you expanded your supposition somewhat more formally, for example, being explicit about all the other assumptions. Just my 2-cents. $\endgroup$
    – cwackers
    Nov 18, 2015 at 16:05

4 Answers 4

2
$\begingroup$

I make some additional assumptions and simplify notations, nothing of which should cause confusion. Suppose for simplicity that data is generated according to $Y = \beta X + \epsilon$, where all variables are $\mathbb R-$valued and $\epsilon$ has zero mean and variance $\sigma^2$. Assume $X$ has the necessary moments. We have $n$ independent copies of the pair $(Y, X)$; $x = [x_1, \dots, x_n]'$ and $y = [y_1,\dots, y_n]'$.

The OLS estimator of $\beta$ is

\begin{align} \hat{\beta} &=y'x / x'x \\ &= (x\beta + e)'x/x'x \\ & = \beta + \frac{e'x}{x'x} \\ &= \beta + \frac{\frac{1}{n}\sum_i \epsilon_i x_i}{\frac{1}{n}\sum_i x_i^2} \end{align}

where $e = [\epsilon_1, \dots, \epsilon_n]'$. The assertion that this approaches $\beta + {\rm Cov}(X, \epsilon) / {\rm Var}(X)$ as $n\to\infty$ is usually in the sense of convergence in probability. According to standard definitions, an estimator is consistent if it converges in probability to the true parameter value, i.e. in this case if $\hat{\beta} \to \beta$ in probability. Here, we had an extra term, in general non-zero, on the right hand side so the estimator is inconsistent.

On the other hand, we say that $\hat{\beta}$ is unbiased if $\mathbb E \hat{\beta} = \beta$. This statement has nothing to do with convergence. All the same, the expectation of the right hand side is again $\beta$ + some term which is not zero in general. Thus, $\hat{\beta}$ is also biased. This answers the first question.

Regarding the second question, notice that bias is usually regarded as a property of estimators, and $\beta$ is unknown so it's not an estimator. Therefore, it does not make sense to speak of the bias of $\beta$. If we are liberal in the usage of bias and let it apply to anything, we see that $\mathbb E \beta = \beta$ so it would be "unbiased".

Without further assumptions on the dependence between $X$ and $\epsilon$ there really isn't any way to tell what the answer to three is, I believe. I should say I did not have time to go through the calculations so I may be wrong.

In light of the discussion under the other answers: If one declares a new definition of consistency in terms of the variance approaching zero, I don't see how inconsistency under the assumption ${\rm Cov}(X, \epsilon) \neq 0$ can be either confirmed or disproved without more assumptions. It's also important to note that consistency in the standard definition says nothing about decreasing variance or decreasing bias. These are separate concepts and should not be mixed up.

$\endgroup$
3
  • $\begingroup$ Thank you for the excellent answer. I have thought more about the third question and edited my original post. Could you please comment on whether omitted-variable bias can be used to compare the variances? $\endgroup$
    – Wise Owl
    Nov 17, 2015 at 20:06
  • $\begingroup$ "consistency in the standard definition says nothing about decreasing variance or decreasing bias". Aren't consistent estimators asymptotically unbiased? $\endgroup$
    – Opt
    Oct 14, 2016 at 22:25
  • $\begingroup$ @Opt no, not in general. They may or may not be. $\endgroup$
    – KOE
    Oct 15, 2016 at 1:55
1
$\begingroup$
  1. Here is a definition quoted from page 30 of this PDF written by the same author (Rubaszek's homepage) of your link's PDF:

    "Consistent estimators: for N -> infinity, the variance converges to 0"

    Consistency appears to be about the variance (of the estimator).

    Bias appears to be about the value (of the estimator).

    I can imagine an estimation procedure where the variance (of the estimator) converges to 0 even though the value (of the estimator) does not converge to its population value, and vice versa.

  2. Theoretically, there are estimation procedures that produce estimators whose variance oscillates forever as N -> infinity.
  3. A population parameter such as $\beta_1$ is taken to be the true value. It cannot be biased. Technically speaking, what is biased is the estimation procedure itself, not the value (such as $\hat{\beta_1}$) that it produces.
$\endgroup$
9
  • $\begingroup$ That definition of "consistent" is at odds with the more usual definition. $\endgroup$
    – Glen_b
    Nov 17, 2015 at 2:27
  • $\begingroup$ Thank you. From your wiki link: "probability of the estimator being arbitrarily close to [true value] converges to one". Isn't that true if and only if the variance converges to zero? I would think so, but stand to be corrected. $\endgroup$
    – cwackers
    Nov 17, 2015 at 2:32
  • $\begingroup$ why the downvote? $\endgroup$
    – cwackers
    Nov 17, 2015 at 2:46
  • $\begingroup$ Both answers confuse different modes of convergence. $\endgroup$
    – KOE
    Nov 17, 2015 at 2:50
  • $\begingroup$ Can you expand on that? $\endgroup$
    – cwackers
    Nov 17, 2015 at 2:53
1
$\begingroup$

When the asymptotic distribution of $\hat \beta_1$ becomes concentrated over the true value of $\beta_1$, it is said to be consistent. This concentration means that in the limit (as the sample size grows), our estimator has zero bias and zero variance as the sampling distribution collapses around the true value. This means that consistency can be loosely thought of as the large-sample equivalent of unbiasedness.

Only estimators can be biased. Similarly, it does not make sense to speak of the variance of $\beta_1$.

$\endgroup$
1
  • 1
    $\begingroup$ This is not correct as written, but may be pedagogically useful. The bias and variance going to zero as the sample grows is sufficient condition for consistency, but it is not strictly necessary. This is arguably the most popular way to show consistency and sort of clarifies the difference from unbiasedness, but it is technically a different type of convergence. Consistency means that $\plim \hat \beta_1 = \beta_1 $, which means that you can get $\hat \beta_1$ arbitrarily close enough to $\beta_1$ with probability of one if you grow the sample sufficiently large. $\endgroup$
    – dimitriy
    Nov 18, 2015 at 1:44
1
$\begingroup$

Here's an example.

Suppose you have a set of n data points from a normal distribution. We assume that the observations are independent.

$$ X_1, X_2, \cdots, X_n \sim N(\mu,\sigma^2) $$

We want to estimate $\mu$. The usual way is a sample average $\overline{x} = \frac{1}{n} \sum_{i=1}^{n} x_i$. The sample average estimator $(\hat{\mu} = \overline{x})$ is consistent and unbiased for the normal distribution. We could define other estimators however, for instance we could just use the last observation for our estimate of the mean $(\hat{\mu} = x_n)$.

The sample average $\overline{x}$ is an unbiased and consistent estimator for $\mu$. The `latest-value' estimator $x_n$ is unbiased but not consistent, the expected value of this estimator is $\mu$ but the variance does not decrease with more observations.

Bias and consistency are not directly related. Neither implies the other.

However if the bias remains non-zero and does not decrease with more observations then the estimator must be inconsistent.

In your supplied document, the author showed that the estimator had a non-zero bias which does not decrease with $n$.

It doesn't make sense to call a parameter or an estimator biased. An estimator is biased with respect to a parameter.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.