Skip to main content
71 votes
Accepted

When is a biased estimator preferable to unbiased one?

Yes. Often it is the case that we are interested in minimizing the mean squared error, which can be decomposed into variance + bias squared. This is an extremely fundamental idea in machine learning, ...
jld's user avatar
  • 20.2k
66 votes
Accepted

Maximum Likelihood Estimators - Multivariate Gaussian

Deriving the Maximum Likelihood Estimators Assume that we have $m$ random vectors, each of size $p$: $\mathbf{X^{(1)}, X^{(2)}, \dotsc, X^{(m)}}$ where each random vectors can be interpreted as an ...
Xavier Bourret Sicotte's user avatar
35 votes
Accepted

Are there parameters where a biased estimator is considered "better" than the unbiased estimator?

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are ...
Peter Flom's user avatar
  • 120k
28 votes
Accepted

Why is an estimator considered a random variable?

Somewhat loosely -- I have a coin in front of me. The value of the next toss of the coin (let's take {Head=1, Tail=0} say) is a random variable. It has some probability of taking the value $1$ ($\...
Glen_b's user avatar
  • 283k
26 votes
Accepted

root-n consistent estimator, but root-n doesn't converge?

What hejseb means is that $\sqrt{n}(\hat\theta-\theta)$ is "bounded in probability", loosely speaking that the probability that $\sqrt{n}(\hat\theta-\theta)$ takes on "extreme" values is "small". Now,...
Christoph Hanck's user avatar
22 votes

Are there parameters where a biased estimator is considered "better" than the unbiased estimator?

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization). As for a mathematical ...
call-in-co's user avatar
  • 1,016
21 votes
Accepted

Use of splines in parameter estimation

I found their code on the Wayback machine and they used the smooth.spline-function in R. The paper points to http://genomine.org/qvalue/results.html for code and ...
Lukas Lohse's user avatar
  • 2,557
20 votes
Accepted

When do maximum likelihood and method of moments produce the same estimators?

A general answer is that an estimator based on a method of moments is not invariant by a bijective change of parameterisation, while a maximum likelihood estimator is invariant. Therefore, they almost ...
Xi'an's user avatar
  • 106k
17 votes
Accepted

How does one explain what an unbiased estimator is to a layperson?

Technically what you're describing when you say that your estimator gets closer to the true value as the sample size grows is (as others have mentioned) consistency, or convergence of statistical ...
dsaxton's user avatar
  • 12.2k
17 votes
Accepted

Why is OLS estimator of AR(1) coefficient biased?

As essentially discussed in the comments, unbiasedness is a finite sample property, and if it held it would be expressed as $$E (\hat \beta ) = \beta$$ (where the expected value is the first moment ...
Alecos Papadopoulos's user avatar
17 votes
Accepted

In some sense, is linear regression an estimate of an estimate of an estimate?

To some extent, you had some very good point. The biggest problem in your interpretation is that you confused the concepts of approximation and estimation. By probability theory, there exists a Borel ...
Zhanxiong's user avatar
  • 18.9k
14 votes

How does one show that there is no unbiased estimator of $\lambda^{-1}$ for a Poisson distribution with mean $\lambda$?

Assume that $g(X_0, \ldots, X_n)$ is an unbiased estimator of $1/\lambda$, that is, $$\sum_{(x_0, \ldots, x_n) \in \mathbb{N}_0^{n+1}} g(x_0, \ldots, x_n) \frac{\lambda^{\sum_{i=0}^n x_i}}{\prod_{i=0}^...
J. Virta's user avatar
  • 481
14 votes
Accepted

What is the oracle property of an estimator?

An oracle knows the truth: it knows the true subset and is willing to act on it. The oracle property is that the asymptotic distribution of the estimator is the same as the asymptotic distribution of ...
user795305's user avatar
  • 2,882
14 votes

Maximum Likelihood Estimators - Multivariate Gaussian

An alternate proof for $\widehat{\Sigma}$ that takes the derivative with respect to $\Sigma$ directly: Picking up with the log-likelihood as above: \begin{eqnarray} \ell(\mu, \Sigma) &=& ...
Eric Kightley's user avatar
14 votes
Accepted

Sample correlation is also a MLE estimator

Given two random variables $X$ and $Y$, their correlation coefficient is: $$\rho_{XY} = \frac{Cov(X,Y)}{\sqrt{Var(X)\cdot Var(Y)}}$$ Where $Cov(X,Y)$ is the covariance of $X$ and $Y$, $Var(X)$ is the ...
mhdadk's user avatar
  • 5,000
14 votes

Why isn't this estimator unbiased?

As indicated in my comment (and then in later answers), the error in the reasoning leading to the apparent paradox is to treat the selected or surviving $X_i$'s that we should denote differently, e.g.,...
Xi'an's user avatar
  • 106k
14 votes
Accepted

Is it wrong to say that a Riemann sum is an unbiased estimate of an integral?

It seems that you are swapping two different concepts here. The concepts are unbiased and consistent, which are properties of an estimator. A sequence of estimators $(T_n)_{n=1}^\infty$ is said to be ...
Lucas Prates's user avatar
  • 1,223
13 votes
Accepted

Is $\overline{X}_n = \frac{X_1 + X_2 + \cdots + X_n}{n}$ an estimator of the mean in general (for random variables with any distribution)?

Estimators are random variables. They exhibit properties that we use to assess their quality, advantages, and disadvantages. So it depends what you mean by "is an estimate of." I can say $\hat{\mu}_0 =...
Mustafa Eisa's user avatar
  • 1,322
12 votes

In some sense, is linear regression an estimate of an estimate of an estimate?

It's not. Consider the problem of estimating a random variable $Y$ using another random variable $X$. We don't estimate random variables, but things about random variables. What would it mean to “...
Tim's user avatar
  • 138k
11 votes

What is the relation between estimator and estimate?

In short: an estimator is a function and an estimate is a value that summarizes an observed sample. An estimator is a function that maps a random sample to the parameter estimate: $$ \hat{\Theta}=t(...
Freeman's user avatar
  • 219
11 votes
Accepted

How to find maximum likelihood estimates of an integer parameter?

You started well by writing down an expression for the likelihood. It is simpler to recognize that $Y,$ being the sum of $N$ independent Normal$(\mu,\sigma^2)$ variables, has a Normal distribution ...
whuber's user avatar
  • 324k
11 votes
Accepted

How to prove $s^2$ is a consistent estimator of $\sigma^2$?

It's a very well known result that : If $X_1, X_2, \cdots, X_n \stackrel{\text{iid}}{\sim} N(\mu,\sigma^2)$ , then $$Z_n = \dfrac{\displaystyle\sum(X_i - \bar{X})^2}{\sigma^2} \sim \chi^2_{n-1}$$ ...
JRC's user avatar
  • 609
11 votes
Accepted

Why isn't this estimator unbiased?

Given that $n_1$ is a random variable (as pointed out already in the comments), the expected value can be computed as $E(\hat\mu)=E_{n_1}[E_{\hat \mu}(\hat\mu|n_1)]$. For the inner expectation, note ...
Christian Hennig's user avatar
11 votes
Accepted

Let $X_1,\dots, X_n$ be random sample from $Bernoulli(p)$. Which estimator is better?

You have been given the MLE as well as the posterior mean under a Beta prior (see e.g. here). As you write (in slightly different notation with $\theta=p$, $k=n\bar X$, $\alpha_0=\alpha$, $\hat \...
Christoph Hanck's user avatar
11 votes

Use of splines in parameter estimation

Edit: In light of Lukas Lohse's answer (which I think should be the accepted one!), my original answer below is misleading. Personally I learned about splines from Tibshirani's books, where he ...
civilstat's user avatar
  • 3,814
10 votes
Accepted

Square of the Sample Mean as estimator of the variance

You have $X_1, X_2, \dots, X_n$ are iid from an unknown distribution with mean (say) $\mu$ and variance (say) $\sigma^2$. $\bar{X}$ is an unbiased estimator of the mean, and thus $E(\bar{X}) = \mu$. ...
Greenparker's user avatar
  • 15.6k
10 votes
Accepted

What's the difference between asymptotic unbiasedness and consistency?

In the related post over at math.se, the answerer takes as given that the definition for asymptotic unbiasedness is $\lim_{n\to \infty} E(\hat \theta_n-\theta) = 0$. Intuitively, I disagree: "...
Alecos Papadopoulos's user avatar
10 votes

Revisiting the Rule of Three

The image below is how I look at confidence intervals. It is an adaptation from an image in the answer to the question 'The basic logic of constructing a confidence interval', which is itself an ...
Sextus Empiricus's user avatar
10 votes

In some sense, is linear regression an estimate of an estimate of an estimate?

A frequentist would use the term "prediction" for what you call "estimation of a random variable". Furthermore, if you don't assume a linear relationship but just use this as ...
Christian Hennig's user avatar
9 votes

Trying to understand an example where unbiased estimators don't exist

I interpret "$B(p)$" to mean a Bernoulli distribution with parameter $p = \Pr(X=1)$ and I suppose $X$ is a single observation from this distribution. Trivial though this situation is, it is ...
whuber's user avatar
  • 324k

Only top scored, non community-wiki answers of a minimum length are eligible