For example, we say that an estimator is unbiased if the expected value of the estimator is the true value of the parameter we're trying to estimate. However, if we already know the true value of the parameter, why are we trying to estimate it? I know there's something I'm missing here, like some theoretical true value that we can find, otherwise I don't see how we can check for unbiasedness, consistency etc.
$\begingroup$
$\endgroup$
11
-
6$\begingroup$ Great question, welcome to the site! We say an estimate is "unbiased" without further qualification if it is unbiased no matter what the true parameter is. Similar for other properties: they must hold for all possible parameter values. (Example: a 90% confidence interval should cover the true value in 90% of samples no matter what that true value is). One thing that is usually left unsaid but will probably help clarify: in the overwhelming majority of models, it is not possible to find unbiased estimators for any possible parameter value; we can only approximate this in large samples. $\endgroup$– John MaddenCommented Dec 11, 2022 at 16:57
-
$\begingroup$ Thanks! So a property like unbiasedness holds for any specific estimator like the OLS for any sample? $\endgroup$– Angelos KoulasCommented Dec 11, 2022 at 22:24
-
3$\begingroup$ "for any sample" no, definitely not for any sample. For some samples, the estimator is going to be a little too high, and for others, a little too low. But if the estimator is unbiased, these cancel out and on average it's on target. $\endgroup$– John MaddenCommented Dec 12, 2022 at 16:20
-
$\begingroup$ @JohnMadden What does it mean for an estimator to be 'a little too high [for a given sample]'? Unbiasedness is a property of an estimator. It is not a function of a sample. The estimate can be too high or too low, but the estimator remains unbiased. $\endgroup$– KukuCommented Dec 15, 2022 at 17:04
-
$\begingroup$ @Kuku estimators are mappings from samples to estimates. For some samples, $\hat{\theta}(\{x_1,\ldots,x_n\})=\theta+\delta$ for positive $\delta$; this is what I mean by a little high for some samples (for other samples, $\delta<0$). If $\hat{\theta}$ is unbiased, $\mathbb{E}\delta=0$. I agree that unbiasedness is not a property of a sample; in fact, this is what I was trying to say in the comment you're replying to (which started with "definitely not for any sample"). $\endgroup$– John MaddenCommented Dec 15, 2022 at 17:15
|
Show 6 more comments
5 Answers
$\begingroup$
$\endgroup$
2
Phil has given a very good answer regarding the unbiasedness of the estimator. But I think your question may actually stem from another confusion:
You seem to mixing up the estimate and the estimator. The estimator is a general rule how to calculate a specific value from a given sample. This sample-specific value is called the estimate.
If you use the estimator $\bar X = \frac{1}{n} \sum_{i=1}^n X_i$ to estimate the unknown population mean $\mu$, then you have an unbiased estimator because $E(\bar X) = \mu$. Applying this estimator to a given sample, you get a specific value $\hat X$ (the sample mean), which is called the estimate. However, this does not mean that $\hat X = \mu$, i.e. the actually estimated mean $\hat X$ for a given sample is not necessarily equal (or even close) to the true value. To get an idea how well we estimated the true value, therefore we usually express our uncertainty of the estimate, e.g. via confidence intervals.
-
$\begingroup$ But if we take the estimated mean from many samples from the same population then we expect that the most frequent value we get is μ if our estimator is unbiased, right? $\endgroup$ Commented Dec 15, 2022 at 9:57
-
1$\begingroup$ @AngelosKoulas technically speaking, $\hat{X}$ will never be exactly $\mu$, but your idea is correct: when you have many samples and one $\hat{X}$ for each sample, most $\hat{X}$ will be close to $\mu$. More precisely, the distribution of $\hat{X}$ will be a $t$-distribution around $\mu$. $\endgroup$– LuckyPalCommented Dec 15, 2022 at 10:18
$\begingroup$
$\endgroup$
Yes, in any real world situation, if you know the value of the parameter, then you don't need to go about trying to estimate it.
One scenario where you know the value of a parameter, and then may want to examine the behavior of a statistic (estimator), is when you are conducting a simulation to see how the statistic behaves in different circumstances. For example, you can ask a computer to, say, start with a population of observations of known distribution and known parameters. You can then make many iterations of sampling this population, using a statistic to estimate some parameter, and see how that statistic behaves in certain situations. For example, what if the sample size is small compared with when it is large ? What if the distribution is skewed compared with when it is symmetric ? ... As a non-statistician, I find this approach useful in assessing how various statistics and tests behave in different circumstances.
answered Dec 11, 2022 at 21:27
$\begingroup$
$\endgroup$
3
Consider i.i.d. sample $X_1, \ldots, X_n \sim N(\mu, 1)$. We assume that all samples are from normal distribution with mean $\mu$, which is unknown, and known variance $1$.
Now, let $\bar X = \frac{1}{n} \sum_{i=1}^n X_i$.
Then $E(\bar X) = \mu$.
So no matter what $\mu$ is, $E(\bar X) = \mu$. It is not necessary to know what $\mu$ is.
-
2$\begingroup$ So essentially when deriving an estimator like the OLS one, and then assessing its unbiasedness, the result (i.e. that OLS is unbiased) holds for any sample, with any true parameter? $\endgroup$ Commented Dec 11, 2022 at 22:23
-
5$\begingroup$ @AngelosKoulas So basically, if you have a data set, it doesn't make sense to talk about whether it's unbiased since it's a fixed numbers. Unbiased comes from random variables. That is, we assume that our samples come from some distribution. $\endgroup$– PhilCommented Dec 12, 2022 at 0:52
-
1$\begingroup$ @AngelosKoulas the idea of unbiasedness is that (speaking very roughly here) if we repeat the same experiment a billion times (ofc impossible in real life) and calculate the statistic for each of the billion experiments and draw a histogram of these billion statistics, then this histogram will be centered around the true value of the parameter. $\endgroup$ Commented Dec 12, 2022 at 17:40
$\begingroup$
$\endgroup$
In practice we don't know the true value of the parameter. However, if we are to use the estimator, we have to know its properties - to be sure that it is a good one (or at least suitable/acceptable under the circumstances.)
What we do usually assume to know is the underlying statistical distribution or, at least, some kind of a model structure (in case of non-parametric estimators.)
$\begingroup$
$\endgroup$
How do we know the true value of a parameter, in order to check estimator properties?
We don't know the true value and we don't know the estimator properties.
One way to deal with it is that we estimate the true value of the parameter, and based on the estimate of the value, we estimate the estimator properties.
This approach is used for instance with the Wald test. This uses the estimated values of the parameters to describe the properties of the estimate.
This approach is an approximation and does not give an exact result.
we say that an estimator is unbiased if the expected value of the estimator is the true value of the parameter we're trying to estimate.
This doesn't require to know the true value. We only need to know that give any true value the expectation of the estimator is equal to that true value.
If for any true value of the parameter the bias is zero, then without knowing the value of the parameter we know that the bias is zero.
However, in cases when the bias is not always zero or a constant, then indeed we can not know the exact bias and we need make an estimate.
answered Dec 12, 2022 at 14:15