Questions tagged [unbiased-estimator]
Refers to an estimator of a population parameter that "hits the true value" on average. That is, a function of the observed data $\hat{\theta}$ is an unbiased estimator of a parameter $\theta$ if $E(\hat{\theta}) = \theta$. The simplest example of an unbiased estimator is the sample mean as an estimator of the population mean.
745
questions
193
votes
3
answers
232k
views
What is the difference between a consistent estimator and an unbiased estimator?
What is the difference between a consistent estimator and an unbiased estimator?
The precise technical definitions of these terms are fairly complicated, and it's difficult to get an intuitive feel ...
- 2,680
85
votes
5
answers
50k
views
How exactly did statisticians agree to using (n-1) as the unbiased estimator for population variance without simulation?
The formula for computing variance has $(n-1)$ in the denominator:
$s^2 = \frac{\sum_{i=1}^N (x_i - \bar{x})^2}{n-1}$
I've always wondered why. However, reading and watching a few good videos about "...
- 14.2k
56
votes
2
answers
6k
views
Intuition behind why Stein's paradox only applies in dimensions $\ge 3$
Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff $n\...
- 1,564
55
votes
5
answers
20k
views
When is a biased estimator preferable to unbiased one?
It's obvious many times why one prefers an unbiased estimator. But, are there any circumstances under which we might actually prefer a biased estimator over an unbiased one?
- 3,893
49
votes
6
answers
10k
views
What can we say about population mean from a sample size of 1?
I am wondering what we can say, if anything, about the population mean, $\mu$ when all I have is one measurement, $y_1$ (sample size of 1). Obviously, we'd love to have more measurements, but we can'...
- 515
43
votes
7
answers
8k
views
Why shouldn't the denominator of the covariance estimator be n-2 rather than n-1?
The denominator of the (unbiased) variance estimator is $n-1$ as there are $n$ observations and only one parameter is being estimated.
$$
\mathbb{V}\left(X\right)=\frac{\sum_{i=1}^{n}\left(X_{i}-\...
- 2,559
35
votes
2
answers
19k
views
Understanding bias-variance tradeoff derivation
I am reading the chapter on the bias-variance tradeoff in The elements of statistical learning and I don't understand the formula on page 29. Let the data arise from a model such that $$ Y = f(x)+\...
- 2,058
32
votes
4
answers
23k
views
Is unbiased maximum likelihood estimator always the best unbiased estimator?
I know for regular problems, if we have a best regular unbiased estimator, it must be the maximum likelihood estimator (MLE). But generally, if we have an unbiased MLE, would it also be the best ...
- 321
32
votes
2
answers
13k
views
Bias correction in weighted variance
For unweighted variance
$$\text{Var}(X):=\frac{1}{n}\sum_i(x_i - \mu)^2$$
there exists the bias corrected sample variance, when the mean was estimated from the same data:
$$\text{Var}(X):=\frac{1}{n-1}...
31
votes
2
answers
7k
views
What is the intuition behind defining completeness in a statistic as being impossible to form an unbiased estimator of $0$ from it?
In classical statistics, there is a definition that a statistic $T$ of a set of data $y_1, \ldots, y_n$ is defined to be complete for a parameter $\theta$ it is impossible to form an unbiased ...
- 2,285
28
votes
3
answers
12k
views
Inference after using Lasso for variable selection
I'm using Lasso for feature selection in a relatively low dimensional setting (n >> p). After fitting a Lasso model, I want to use the covariates with nonzero coefficients to fit a model with no ...
- 2,533
27
votes
2
answers
14k
views
What is importance sampling?
I'm trying to learn reinforcement learning and this topic is really confusing to me. I have taken an introduction to statistics, but I just couldn't understand this topic intuitively.
- 461
26
votes
6
answers
15k
views
Intuitive understanding of the difference between consistent and asymptotically unbiased [duplicate]
I am trying to to get an intuitive understanding and feel for the difference and practical difference between the term consistent and asymptotically unbiased. I know their mathematical/statistical ...
- 10.8k
25
votes
3
answers
3k
views
Unbiased estimation of covariance matrix for multiply censored data
Chemical analyses of environmental samples are often censored below at reporting limits or various detection/quantitation limits. The latter can vary, usually in proportion to the values of other ...
- 309k
24
votes
5
answers
8k
views
Why are we using a biased and misleading standard deviation formula for $\sigma$ of a normal distribution?
It came as a bit of a shock to me the first time I did a normal distribution Monte Carlo simulation and discovered that the mean of $100$ standard deviations from $100$ samples, all having a sample ...
- 12.3k
23
votes
2
answers
2k
views
Shrunken $r$ vs unbiased $r$: estimators of $\rho$
There has been some confusion in my head about two types of estimators of the population value of Pearson correlation coefficient.
A. Fisher (1915) showed that for bivariate normal population ...
- 55.1k
22
votes
4
answers
24k
views
What does "unbiasedness" mean?
What does it mean to say that "the variance is a biased estimator".
What does it mean to convert a biased estimate to an unbiased estimate through a simple formula. What does this conversion do ...
- 2,807
21
votes
4
answers
3k
views
Flaws in Frequentist Inference
I have problem to understanding the following example.
(1) After the next day that the glitch discovered what can tell about the observation? $X_i\nsim N(\mu,1)$ or just $X_i\sim N(\mu_2,1)$. Some ...
- 1,229
21
votes
2
answers
2k
views
Is there an unbiased estimator of the Hellinger distance between two distributions?
In a setting where one observes $X_1,\ldots,X_n$ distributed from a distribution with density $f$, I wonder if there is an unbiased estimator (based on the $X_i$'s) of the Hellinger distance to ...
- 99.7k
20
votes
4
answers
3k
views
OLS is BLUE. But what if I don't care about unbiasedness and linearity?
The Gauss-Markov theorem tells us that the OLS estimator is the best linear unbiased estimator for the linear regression model.
But suppose I don't care about linearity and unbiasedness. Then is ...
20
votes
2
answers
635
views
Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict")
This question references Galit Shmueli's paper "To Explain or to Predict".
Specifically, in section 1.5, "Explaining and Prediction are Different", Professor Shmueli writes:
In explanatory ...
- 34.9k
18
votes
5
answers
1k
views
Why do US and UK Schools Teach Different methods of Calculating the Standard Deviation?
As I understand UK Schools teach that the Standard Deviation is found using:
whereas US Schools teach:
(at a basic level anyway).
This has caused a number of my students problems in the past as ...
- 526
18
votes
4
answers
2k
views
In regression, why not use regularization by default?
I remember reading somewhere in another post about the different viewpoints between people from statistics and from machine learning or neural networks, where one user was mentioning this idea as an ...
- 1,241
18
votes
3
answers
7k
views
Why does one have to use REML (instead of ML) for choosing among nested var-covar models?
Various descriptions on model selection on random effects of Linear Mixed Models instruct to use REML. I know difference between REML and ML at some level, but I don't understand why REML should be ...
- 1,394
17
votes
2
answers
959
views
For which distributions is there a closed-form unbiased estimator for the standard deviation?
For the normal distribution, there is an unbiased estimator of the standard deviation given by:
$$\hat{\sigma}_\text{unbiased} = \frac{\Gamma(\frac{n-1}{2})}{\Gamma(\frac{n}{2})} \sqrt{\frac{1}{2}\...
- 22.1k
16
votes
1
answer
4k
views
Other unbiased estimators than the BLUE (OLS solution) for linear models
For a linear model the OLS solution provides the best linear unbiased estimator for the parameters.
Of course we can trade in a bias for lower variance, e.g. ridge regression. But my question is ...
- 3,641
16
votes
1
answer
8k
views
Bias of maximum likelihood estimators for logistic regression
I would like to understand a couple of fact on maximum likelihood estimators (MLEs) for logistic regressions.
Is it true that, in general, the MLE for logistic regression is biased? I would say "yes"....
- 660
16
votes
1
answer
7k
views
What's the difference between asymptotic unbiasedness and consistency?
Does each imply the other? If not, does one imply the other? Why/why not?
This issue came up in response to a comment on an answer I posted here.
Although google searching the relevant terms didn't ...
16
votes
1
answer
870
views
An unbiased estimator of the ratio of two regression coefficients?
Suppose you fit a linear/logistic regression $g(y) = a_0 + a_1\cdot x_1 + a_2\cdot x_2$, with the aim of an unbiased estimate of $\frac{a_1}{a_2}$. You are very confident that both $a_1$ and $a_2$ ...
- 340
15
votes
3
answers
2k
views
Are there parameters where a biased estimator is considered "better" than the unbiased estimator? [duplicate]
A perfect estimator would be accurate (unbiased) and precise (good estimation even with small samples).
I never really thought of the question of precision but only the one of accuracy (as I did in ...
- 4,892
15
votes
2
answers
6k
views
Estimating parameters of a normal distribution: median instead of mean?
The common approach for estimating the parameters of a normal distribution is to use the mean and the sample standard deviation / variance.
However, if there are some outliers, the median and the ...
- 3,188
15
votes
1
answer
4k
views
Is the mean (Bayesian) posterior estimate of $\theta$ a (Frequentist) unbiased estimator of $\theta$?
I am wondering about the different ways that Bayesian and Frequentist statistic connect with each other.
I recalled that the Maximum Likelihood estimate of a parameter $\theta$ is not necessarily an ...
- 2,439
15
votes
1
answer
5k
views
How does one show that there is no unbiased estimator of $\lambda^{-1}$ for a Poisson distribution with mean $\lambda$?
Suppose that $ X_{0},X_{1},\ldots,X_{n} $ are i.i.d. random variables that follow the Poisson distribution with mean $ \lambda $. How can I prove that there is no unbiased estimator of the quantity $ \...
- 251
14
votes
4
answers
4k
views
How does one explain what an unbiased estimator is to a layperson?
Suppose $\hat{\theta}$ is an unbiased estimator for $\theta$. Then of course, $\mathbb{E}[\hat{\theta} \mid \theta] = \theta$.
How does one explain this to a layperson? In the past, what I have said ...
- 4,213
14
votes
2
answers
1k
views
Unbiased estimator of exponential of measure of a set?
Suppose we have a (measurable and suitably well-behaved) set $S\subseteq B\subset\mathbb R^n$, where $B$ is compact. Moreover, suppose we can draw samples from the uniform distribution over $B$ wrt ...
- 799
14
votes
5
answers
488
views
Different usage of the term "Bias" in stats/machine learning
I think I've seen about 4 different usages of the word "bias" in stats/ML, and all these usages seem to be non-related. I just wanted to get clarification that the usages are indeed non-...
- 404
14
votes
2
answers
1k
views
Model for population density estimation
A database of (population, area, shape) can be used to map population density by assigning a constant value of population/area to each shape (which is a polygon such as a Census block, tract, county, ...
- 309k
14
votes
1
answer
2k
views
Does a median-unbiased estimator minimize mean absolute deviance?
This is a follow-up but also a different question of my previous one.
I read on Wikipedia that "A median-unbiased estimator minimizes the risk with respect to the absolute-deviation loss function, as ...
- 393
14
votes
1
answer
959
views
Biased estimator for regression achieving better results than unbiased one in Error In Variables Model
I am working on some syntatic data for Error In Variable model for some research. Currently I have a single independent variable, and I am assuming I know the variance for the true value of the ...
- 191
13
votes
2
answers
25k
views
Correct equation for weighted unbiased sample covariance
I'm looking for the correct equation to compute the weighted unbiased sample covariance. Internet sources are quite rare on this theme and they all use different equations.
The most likely equation I'...
- 717
13
votes
4
answers
1k
views
Unbiased estimator for the smaller of two random variables
Suppose $X \sim \mathcal{N}(\mu_x, \sigma^2_x)$ and $Y \sim \mathcal{N}(\mu_y, \sigma^2_y)$
I am interested in $z = \min(\mu_x, \mu_y)$. Is there an unbiased estimator for $z$?
The simple estimator ...
- 131
12
votes
4
answers
1k
views
Why isn't this estimator unbiased?
Suppose we have a IID sample $X_1, X_2, \cdots, X_n$ with each $X_i$ distributed as $\mathcal{N}(\mu, \sigma^2)$. Now suppose we construct (a rather peculiar) estimator for the mean $\mu$: we only ...
- 603
12
votes
5
answers
3k
views
Does the biased estimator always have less variance than unbiased one?
Suppose I am estimating one of the parameter. Now if we plot the biased estimator of that and unbiased estimator of that can we say for sure that biased one has less variance than unbiased one always.
...
- 279
12
votes
2
answers
7k
views
Why is bias equal to zero for OLS estimator with respect to linear regression?
I understand the concept of bias-variance tradeoff. Bias based on my understanding, represents the error because of using a simple classifer(eg: linear) to capture a complex non-linear decision ...
- 5,493
11
votes
2
answers
17k
views
Using MSE instead of log-loss in logistic regression
Suppose we replace the loss function of the logistic regression (which is normally log-likelihood) with the MSE. That is, still have log odds ratio be a linear function of the parameters, but minimize ...
- 1,584
11
votes
1
answer
1k
views
Should the standard deviation be corrected in a Student's T test?
Using the Student's T test, T-Critical is calculated via:
$t = \frac{\bar{X} - \mu_{0}}{s / \sqrt{n}}$
Looking at Wikipedia article on the unbiased Estimation of the standard deviation, there ...
- 390
11
votes
2
answers
4k
views
Bias / variance tradeoff math
I understand the matter in the underfitting / overfitting terms but I still struggle to grasp the exact math behind it. I've checked several sources (here, here, here, here and here) but I still don't ...
- 291
11
votes
2
answers
3k
views
Degrees of Freedom In Sample Variance
Recall the formula for sample variance $$s_{n - 1}^2 = \dfrac{1}{n -1} \sum_{i = 1}^n (\bar{x} - x_i)^2,$$ where $\bar{x}$ is the sample mean. There are many proofs for why $s_{n - 1}^2$ is an ...
11
votes
1
answer
3k
views
When is having an unbiased estimator important?
We have a few questions and answers about when one would prefer a biased estimate over a unbiased one, but I have not found anything on the reverse question:
In what situations is it important to ...
- 34.9k
11
votes
2
answers
830
views
Bayes estimator are immune to selection Bias
Are Bayes estimators immune to selection bias?
Most papers that discuss estimation in high dimension, e.g., whole genome sequence data, will often raise the issue of selection bias. Selection bias ...
- 811