6
$\begingroup$

Consider a sample of 34 pairs of values $(x,y)$ for the regression equation $$ y_{i}=\alpha + \beta x_{i} + \epsilon_i . $$

Using linear regression (OLS), I got the estimate $\hat{\beta}=2.3$.

  • What is the probability that $\beta>\hat{\beta}$
  • or $ \beta > 2.3$ in this case?

EDIT

As @TooTone mentions in the comments, the probability that $\beta>2.3$ is either $0$ or $1$, as the underlying $\beta$ is fixed (but unknown). My follow-up question is this:

  • What is the probability that $\hat{\beta}>2.3$?

EDIT 2

  1. A simple linear regression model $y_{i}=\alpha +\beta x_{i}+\varepsilon_{_i}$ is set up to look at a relationship between x and y. The estimated $\beta$ ($\hat{\beta }$) with OLS is 2.3 and the 2.3 is used in next stage calculation, say risk management. The larger of it is used the worse things are going to be. Then, I am thinking why I use 2.3, of which the reason seems obvious – because it comes from the regression model. Then, I ask myself what is probability the factor is larger than 2.3.

  2. I did some read and my feeling was the probability of the ($\hat{\beta }$) larger than 2.3 is exactly ½ because t distribution $t=\frac{\hat{\beta }-\beta }{SE_{\hat{\beta }}}$. (e.g http://en.wikipedia.org/wiki/Simple_linear_regression). The finding startled me a bit – it likes throwing a coin. While I am writing this I look at the t distribution equation again and realize t value is centered at “true value“ $\beta$. 2.3 is not the true value of $\beta$. Is it a fair question to ask as there is no way to know the “true” $\beta$.

One more thing to add. According to $t=\frac{\hat{\beta }-\beta }{SE_{\hat{\beta }}}$. Prob$(\hat{\beta }>\beta)=1/2$, as @Macro has pointed out.

$\endgroup$
  • 6
    $\begingroup$ Are the errors IID and normally distributed? If so, $\hat \beta$ is normally distributed with mean $\beta$, so the probability is $1/2$. $\endgroup$ – Macro Jul 17 '13 at 15:13
  • 5
    $\begingroup$ @user603, if the errors are not normally distributed then I don't know the finite sample distribution of $\hat \beta$. Gauss-Markov only tells me that $\hat \beta$ is the BLUE. So, it's not obvious to me that $P(\hat \beta > \beta)$ is always $1/2$. If you can show this, please post an answer! $\endgroup$ – Macro Jul 17 '13 at 15:17
  • 3
    $\begingroup$ On a different note, the probability that $\beta > 2.3$ is either $0$ or $1$, because the underlying $\beta$ is fixed (albeit unknown). It only makes sense to ask whether $\beta > \hat\beta$, because $\hat\beta$ is random. $\endgroup$ – TooTone Jul 17 '13 at 15:33
  • 6
    $\begingroup$ @TooTone That was a nice guess, but it's not correct. You fell victim to the symmetry of your $x$ variable, which (when standardized) is identical to its set of negatives. That implies the median of $\hat{\beta}$ equals $\beta$. For an illustrative counterexample do this: x <- c(1:33 / 10^6, 1); n <- 500; y <- matrix(exp(rnorm(length(x) * n) - 1/2), nrow=n); sim <- apply(y, 1, function(z) coef(lm(z ~ x))[2]); p.hat <- sum(sim > 0) / n; (p.hat - 1/2) / sqrt(p.hat * (1 - p.hat) / n). Generate Normal variates and re-run to compare. $\endgroup$ – whuber Jul 17 '13 at 15:44
  • 2
    $\begingroup$ The edit is still confusing: $\hat{\beta}$ is what it is, so given the value of this estimate, $\Pr(\hat{\beta}\gt 2.3)$ is zero. Before you did the regression, it made sense to ask about this probability: but obtaining an answer is impossible, because the probability depends in unknown ways on the value of $\beta$ (which is what you're trying to estimate in the first place!). One question that does make sense, and sometimes can be answered, concerns $\Pr(\hat{\beta}\gt \beta)$. Although you don't know $\beta$, $\hat{\beta}$ has a definite mathematical relationship with $\beta$. $\endgroup$ – whuber Jul 17 '13 at 20:09
2
$\begingroup$

The OLS (or any other) estimator, $\hat \beta$, is a random variable. Namely, it is a real-valued function. It takes as input the sample data and produces a real number. This real number is the sample-specific estimate. The habit of using the same symbol to denote the function and a specific value of it can become confusing, as some comment showed. Being a random variable, $\hat \beta$ has a proper distribution function, $F_{\hat \beta}\left(\hat \beta\right)$. Then it is valid to ask questions like $$P(\hat \beta \gt c) = ?\, \Rightarrow 1-F_{\hat \beta}\left(c\right) =?,\;c\in \Bbb R$$

I said "it is valid", I didn't say it will give you a tangible result. And this is because this distribution will involve the unknown parameter $\beta$, which remains unknown despite our estimation efforts. So even if you are in a position to specify the distribution as belonging to some family (like normal or Student's t or whatever), you will not be able to get a specific numerical value for the above probability you are looking for, because some parameter of this distribution will be unknown.

Moreover, any specific estimate, like 2.3 is just a point in the support of the density of $\hat \beta$. We have no way of knowing whether it is the "true value" of $\beta$- this would be equivalent to believe that with one sample we hit dead-center and uncovered the true value of $\beta$. So even if we assume that the distribution of the estimator is symmetric, we don't know if the specific estimate is the expected value=median of its distribution (as someone commented). So the statement $P(\hat \beta \gt 2.3) = \frac12$ is wrong, remembering that $2.3= \hat \beta\left(\text{sample(j)}\right)$. The statement $P(\hat \beta \gt \beta) = \frac12$ is correct if we assume a) that $\hat \beta$ is an unbiased estimator of $\beta$ and b) that $\hat \beta$ has a symmetric distribution.

If our sample $j$ is very large, and we assume/accept/prove that our estimator is consistent, then some weight can be given to the argument that $\hat \beta\left(\text{sample(j)}\right)\approx \beta$ - this is the essence of consistency, and this is why consistency is "informally" considered a more important property of estimators than unbiasedness: the desirable consequences of consistency can be at least partially "bestowed" upon a single estimate, if the sample size is large (and eventually we are obtaining more and more large samples). The desirable consequences of unbiasedness need many samples and many estimates for them to emerge. If we are in a position to estimate many different samples, then we will be able to obtain many different estimates, and then take their mean value as the "true value" of $\beta$, if, again, we have reasons to believe that $\hat \beta$ is an unbiased estimator.

Probabilistic questions about $\beta$ can only be asked in a Beaysian framework. Here, we model the unknown parameter itself as a random variable to reflect our ignorance about it. In this context there is no distinction between a fixed unknown parameter $\beta$ and a function $\hat \beta$ that tries to estimate it, and so it makes sense to ask $P(\beta\gt c|sample) = ?\, $

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.