Questions tagged [unbiased-estimator]
Refers to an estimator of a population parameter that "hits the true value" on average. That is, a function of the observed data $\hat{\theta}$ is an unbiased estimator of a parameter $\theta$ if $E(\hat{\theta}) = \theta$. The simplest example of an unbiased estimator is the sample mean as an estimator of the population mean.
139
questions
48
votes
7
answers
9k
views
Why shouldn't the denominator of the covariance estimator be n-2 rather than n-1?
The denominator of the (unbiased) variance estimator is $n-1$ as there are $n$ observations and only one parameter is being estimated.
$$
\mathbb{V}\left(X\right)=\frac{\sum_{i=1}^{n}\left(X_{i}-\...
- 2,729
196
votes
3
answers
245k
views
What is the difference between a consistent estimator and an unbiased estimator?
What is the difference between a consistent estimator and an unbiased estimator?
The precise technical definitions of these terms are fairly complicated, and it's difficult to get an intuitive feel ...
- 2,740
24
votes
5
answers
9k
views
Why are we using a biased and misleading standard deviation formula for $\sigma$ of a normal distribution?
It came as a bit of a shock to me the first time I did a normal distribution Monte Carlo simulation and discovered that the mean of $100$ standard deviations from $100$ samples, all having a sample ...
- 13.1k
86
votes
5
answers
53k
views
How exactly did statisticians agree to using (n-1) as the unbiased estimator for population variance without simulation?
The formula for computing variance has $(n-1)$ in the denominator:
$s^2 = \frac{\sum_{i=1}^N (x_i - \bar{x})^2}{n-1}$
I've always wondered why. However, reading and watching a few good videos about "...
- 14.7k
33
votes
3
answers
8k
views
What is the intuition behind defining completeness in a statistic as being impossible to form an unbiased estimator of $0$ from it?
In classical statistics, there is a definition that a statistic $T$ of a set of data $y_1, \ldots, y_n$ is defined to be complete for a parameter $\theta$ it is impossible to form an unbiased ...
- 2,365
57
votes
5
answers
21k
views
When is a biased estimator preferable to unbiased one?
It's obvious many times why one prefers an unbiased estimator. But, are there any circumstances under which we might actually prefer a biased estimator over an unbiased one?
- 4,353
50
votes
6
answers
11k
views
What can we say about population mean from a sample size of 1?
I am wondering what we can say, if anything, about the population mean, $\mu$ when all I have is one measurement, $y_1$ (sample size of 1). Obviously, we'd love to have more measurements, but we can'...
- 525
18
votes
3
answers
8k
views
Why does one have to use REML (instead of ML) for choosing among nested var-covar models?
Various descriptions on model selection on random effects of Linear Mixed Models instruct to use REML. I know difference between REML and ML at some level, but I don't understand why REML should be ...
- 1,436
16
votes
1
answer
9k
views
Bias of maximum likelihood estimators for logistic regression
I would like to understand a couple of fact on maximum likelihood estimators (MLEs) for logistic regressions.
Is it true that, in general, the MLE for logistic regression is biased? I would say "yes"....
- 670
12
votes
2
answers
4k
views
Bias / variance tradeoff math
I understand the matter in the underfitting / overfitting terms but I still struggle to grasp the exact math behind it. I've checked several sources (here, here, here, here and here) but I still don't ...
- 301
16
votes
1
answer
7k
views
What's the difference between asymptotic unbiasedness and consistency?
Does each imply the other? If not, does one imply the other? Why/why not?
This issue came up in response to a comment on an answer I posted here.
Although google searching the relevant terms didn't ...
28
votes
6
answers
18k
views
Intuitive understanding of the difference between consistent and asymptotically unbiased [duplicate]
I am trying to to get an intuitive understanding and feel for the difference and practical difference between the term consistent and asymptotically unbiased. I know their mathematical/statistical ...
- 11.4k
28
votes
3
answers
14k
views
Inference after using Lasso for variable selection
I'm using Lasso for feature selection in a relatively low dimensional setting (n >> p). After fitting a Lasso model, I want to use the covariates with nonzero coefficients to fit a model with no ...
- 2,652
25
votes
3
answers
3k
views
Unbiased estimation of covariance matrix for multiply censored data
Chemical analyses of environmental samples are often censored below at reporting limits or various detection/quantitation limits. The latter can vary, usually in proportion to the values of other ...
- 323k
20
votes
2
answers
775
views
Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict")
This question references Galit Shmueli's paper "To Explain or to Predict".
Specifically, in section 1.5, "Explaining and Prediction are Different", Professor Shmueli writes:
In explanatory ...
- 35.7k
14
votes
4
answers
4k
views
How does one explain what an unbiased estimator is to a layperson?
Suppose $\hat{\theta}$ is an unbiased estimator for $\theta$. Then of course, $\mathbb{E}[\hat{\theta} \mid \theta] = \theta$.
How does one explain this to a layperson? In the past, what I have said ...
- 5,007
9
votes
1
answer
4k
views
For the binomial distribution, why does no unbiased estimator exist for $1/p$?
Suppose that $X$ ~ $Binomial(n,p)$ for $0 < p < 1$
Why does no unbiased estimator exist for $1/p$?
My approach:
We try to find the structure of $E_p(U(x))$, where $U(x)$ is any estimator of $...
- 2,077
5
votes
2
answers
2k
views
Proving OLS unbiasedness without conditional zero error expectation?
The OLS estimate $b$ is equal to $(X^TX)^{-1}X^Ty$ for the linear regression model. If we assume that $E(\epsilon|X)=0$ then it is easy to prove simply by taking the conditional expectation, of $b$ ...
- 2,809
3
votes
1
answer
444
views
MAE regression gives biased regression parameters for symmetric error?
Consider a linear model,
$$
y_i = \beta_0 + \beta_1x_{1i} + \beta_2x_{2i} + \epsilon_i.
$$
From the Gauss-Markov theorem, I know that, under nice conditions, the $\hat{\beta}_{OLS}=(X^TX)^{-1}X^Ty$ ...
- 62.4k
24
votes
4
answers
25k
views
What does "unbiasedness" mean?
What does it mean to say that "the variance is a biased estimator".
What does it mean to convert a biased estimate to an unbiased estimate through a simple formula. What does this conversion do ...
- 3,137
58
votes
2
answers
6k
views
Intuition behind why Stein's paradox only applies in dimensions $\ge 3$
Stein's Example shows that the maximum likelihood estimate of $n$ normally distributed variables with means $\mu_1,\ldots,\mu_n$ and variances $1$ is inadmissible (under a square loss function) iff $n\...
- 1,594
37
votes
2
answers
24k
views
Understanding bias-variance tradeoff derivation
I am reading the chapter on the bias-variance tradeoff in The elements of statistical learning and I don't understand the formula on page 29. Let the data arise from a model such that $$ Y = f(x)+\...
- 2,098
33
votes
2
answers
14k
views
Bias correction in weighted variance
For unweighted variance
$$\text{Var}(X):=\frac{1}{n}\sum_i(x_i - \mu)^2$$
there exists the bias corrected sample variance, when the mean was estimated from the same data:
$$\text{Var}(X):=\frac{1}{n-1}...
- 42.4k
17
votes
5
answers
791
views
Different usage of the term "Bias" in stats/machine learning
I think I've seen about 4 different usages of the word "bias" in stats/ML, and all these usages seem to be non-related. I just wanted to get clarification that the usages are indeed non-...
- 434
15
votes
1
answer
5k
views
How does one show that there is no unbiased estimator of $\lambda^{-1}$ for a Poisson distribution with mean $\lambda$?
Suppose that $ X_{0},X_{1},\ldots,X_{n} $ are i.i.d. random variables that follow the Poisson distribution with mean $ \lambda $. How can I prove that there is no unbiased estimator of the quantity $ \...
- 251
17
votes
1
answer
5k
views
Other unbiased estimators than the BLUE (OLS solution) for linear models
For a linear model the OLS solution provides the best linear unbiased estimator for the parameters.
Of course we can trade in a bias for lower variance, e.g. ridge regression. But my question is ...
- 3,711
17
votes
1
answer
5k
views
Is the mean (Bayesian) posterior estimate of $\theta$ a (Frequentist) unbiased estimator of $\theta$?
I am wondering about the different ways that Bayesian and Frequentist statistic connect with each other.
I recalled that the Maximum Likelihood estimate of a parameter $\theta$ is not necessarily an ...
- 2,809
14
votes
2
answers
4k
views
Degrees of Freedom In Sample Variance
Recall the formula for sample variance $$s_{n - 1}^2 = \dfrac{1}{n -1} \sum_{i = 1}^n (\bar{x} - x_i)^2,$$ where $\bar{x}$ is the sample mean. There are many proofs for why $s_{n - 1}^2$ is an ...
14
votes
2
answers
27k
views
Correct equation for weighted unbiased sample covariance
I'm looking for the correct equation to compute the weighted unbiased sample covariance. Internet sources are quite rare on this theme and they all use different equations.
The most likely equation I'...
- 757
13
votes
2
answers
18k
views
Using MSE instead of log-loss in logistic regression
Suppose we replace the loss function of the logistic regression (which is normally log-likelihood) with the MSE. That is, still have log odds ratio be a linear function of the parameters, but minimize ...
- 1,664
3
votes
2
answers
1k
views
UMVUE for Bernoulli
Let $X_1,..,X_n$ be independent and $Bin(1,\theta)$ distributed. I would like to find the UMVUE for $\phi(\theta)=\theta^3$. I have a complete and sufficient statistic in $T=\sum_iX_i$, and a unbiased ...
- 903
23
votes
2
answers
2k
views
Shrunken $r$ vs unbiased $r$: estimators of $\rho$
There has been some confusion in my head about two types of estimators of the population value of Pearson correlation coefficient.
A. Fisher (1915) showed that for bivariate normal population ...
- 57.6k
20
votes
4
answers
3k
views
OLS is BLUE. But what if I don't care about unbiasedness and linearity?
The Gauss-Markov theorem tells us that the OLS estimator is the best linear unbiased estimator for the linear regression model.
But suppose I don't care about linearity and unbiasedness. Then is ...
17
votes
3
answers
7k
views
Estimating parameters of a normal distribution: median instead of mean?
The common approach for estimating the parameters of a normal distribution is to use the mean and the sample standard deviation / variance.
However, if there are some outliers, the median and the ...
- 3,348
11
votes
2
answers
2k
views
Unbiased estimator for AR($p$) model
Consider an AR($p$) model (assuming zero mean for simplicity):
$$ x_t = \varphi_1 x_{t-1} + \dotsc + \varphi_p x_{t-p} + \varepsilon_t $$
The OLS estimator (equivalent to the conditional maximum ...
- 67.6k
10
votes
1
answer
1k
views
Optimal importance sampling with ratio estimator
We want to approximate the following expectation:
$$\mathbb{E}[h(x)] = \int h(x)\pi(x) dx$$
Where $h(x)$ is an arbitrary function and $\pi(x)$ is a distribution, also for simplicity, let's assume ...
- 1,248
10
votes
1
answer
477
views
Are unbiased efficient estimators stochastically dominant over other (median) unbiased estimators?
General description
Does an efficient estimator (which has sample variance equal to the Cramér–Rao bound) maximize the probability for being close to the true parameter $\theta$?
Say we compare the ...
- 78.5k
8
votes
3
answers
3k
views
Consistent unbiased estimator for the location parameter of $\mathcal{Cauchy} (\theta, 1)$
Given Cauchy distribution with pdf $p(x) = \frac{1}{\pi ((x - \theta)^2 + 1)}$
how can I find a consistent unbiased estimator for $\theta$?
My reasoning so far
Tried MLE, but there seems to be no ...
- 181
7
votes
1
answer
2k
views
Biased prediction (overestimation) for xgboost
I run xgboost and elastic-net on the same dataset for a classification problem, say we have
...
- 469
6
votes
2
answers
5k
views
How do I use the standard regression assumptions to prove that $\hat{\sigma}^2$ is an unbiased estimator of $\sigma^2$?
I'm working through an econometrics textbook and it's proving that
$$
\sigma^2 = E(\hat{\sigma}^2) = \frac{SSR}{n-2}
$$
I followed the proof (an example of which is shown on talkstats) until it ...
- 621
5
votes
2
answers
3k
views
Finding UMVUE of a function of parameter belonging to Poisson distribution
Let $X_1, ..., X_n$ be iid from the Poisson ($\theta$) distribution.
I have proven that $T = \sum_{i=1}^{n} x_i$ is the complete and sufficient statistic and it has a Poisson($n\theta$) distribution....
- 323
5
votes
2
answers
1k
views
Finding UMVUE for a function of a Bernoulli parameter
Given $m$ i.i.d. Bernoulli( $\theta$ ) r.v.s $X_{1}, X_{2}, \ldots, X_{m},$ I'm interested in finding the UMVUE of $(1-\theta)^{1/k}$, when $k$ is a positive integer. .
I know $\sum X_{i}$ is a ...
- 214
5
votes
1
answer
2k
views
Unbiased Estimator of the Variance of the Sample Variance
At Mathematics Stack Exchange, user940 provided a general formula to calculate the variance of the sample variance based on the fourth central moment $\mu_4$ and the population variance $\...
- 323
1
vote
2
answers
1k
views
Definition of the bias of an estimator
I'm quite confused about the definition of the bias of an estimator.
Suppose we have unknown distribution $P(x, \theta)$, and construct the
estimator $\hat{\theta}$ that maps the observed data sample ...
- 11
33
votes
4
answers
26k
views
Is unbiased maximum likelihood estimator always the best unbiased estimator?
I know for regular problems, if we have a best regular unbiased estimator, it must be the maximum likelihood estimator (MLE). But generally, if we have an unbiased MLE, would it also be the best ...
- 331
27
votes
2
answers
14k
views
What is importance sampling?
I'm trying to learn reinforcement learning and this topic is really confusing to me. I have taken an introduction to statistics, but I just couldn't understand this topic intuitively.
- 461
22
votes
5
answers
1k
views
Why do US and UK Schools Teach Different methods of Calculating the Standard Deviation?
As I understand UK Schools teach that the Standard Deviation is found using:
whereas US Schools teach:
(at a basic level anyway).
This has caused a number of my students problems in the past as ...
- 576
18
votes
3
answers
1k
views
For which distributions is there a closed-form unbiased estimator for the standard deviation?
For the normal distribution, there is an unbiased estimator of the standard deviation given by:
$$\hat{\sigma}_\text{unbiased} = \frac{\Gamma(\frac{n-1}{2})}{\Gamma(\frac{n}{2})} \sqrt{\frac{1}{2}\...
- 23.4k
15
votes
2
answers
1k
views
Model for population density estimation
A database of (population, area, shape) can be used to map population density by assigning a constant value of population/area to each shape (which is a polygon such as a Census block, tract, county, ...
- 323k
13
votes
4
answers
1k
views
Unbiased estimator for the smaller of two random variables
Suppose $X \sim \mathcal{N}(\mu_x, \sigma^2_x)$ and $Y \sim \mathcal{N}(\mu_y, \sigma^2_y)$
I am interested in $z = \min(\mu_x, \mu_y)$. Is there an unbiased estimator for $z$?
The simple estimator ...
- 131