The Stack Overflow podcast is back! Listen to an interview with our new CEO.
126

To define the two terms without using too much technical language: An estimator is consistent if, as the sample size increases, the estimates (produced by the estimator) "converge" to the true value of the parameter being estimated. To be slightly more precise - consistency means that, as the sample size increases, the sampling distribution of the ...


51

Yes. Often it is the case that we are interested in minimizing the mean squared error, which can be decomposed into variance + bias squared. This is an extremely fundamental idea in machine learning, and statistics in general. Frequently we see that a small increase in bias can come with a large enough reduction in variance that the overall MSE decreases. A ...


34

One example is estimates from ordinary least squares regression when there is collinearity. They are unbiased but have huge variance. Ridge regression on the same problem yields estimates that are biased but have much lower variance. E.g. install.packages("ridge") library(ridge) set.seed(831) data(GenCont) ridgemod <- linearRidge(Phenotypes ~ ., data = ...


24

Consistency of an estimator means that as the sample size gets large the estimate gets closer and closer to the true value of the parameter. Unbiasedness is a finite sample property that is not affected by increasing sample size. An estimate is unbiased if its expected value equals the true parameter value. This will be true for all sample sizes and is ...


24

Deriving the Maximum Likelihood Estimators Assume that we have $m$ random vectors, each of size $p$: $\mathbf{X^{(1)}, X^{(2)},...,X^{(m)}}$ where each random vectors can be interpreted as an observation (data point) across $p$ variables. If each $\mathbf{X}^{(i)}$ are i.i.d. as multivariate Gaussian vectors: $$ \mathbf{X^{(i)}} \sim \mathcal{N}_p(\mu, \...


23

Point estimates and confidence intervals are for parameters that describe the distribution, e.g. mean or standard deviation. But unlike other sample statistics like the sample mean and the sample standard deviation the p-value is not an useful estimator of an interesting distribution parameter. Look at the answer by @whuber for technical details. The p-...


21

Yes, it could be (and has been) argued that a p-value is a point estimate. In order to identify whatever property of a distribution a p-value might estimate, we would have to assume it is asymptotically unbiased. But, asymptotically, the mean p-value for the null hypothesis is $1/2$ (ideally; for some tests it might be some other nonzero number) and for any ...


20

Yes there are plenty of cases; you're beating around the bush that is the topic of Bias-Variance tradeoff (in particular, the graphic to the right is a good visualization). As for a mathematical example, I am pulling the following example from the excellent Statistical Inference by Casella and Berger to show that a biased estimator has lower Mean Squared ...


20

Somewhat loosely -- I have a coin in front of me. The value of the next toss of the coin (let's take {Head=1, Tail=0} say) is a random variable. It has some probability of taking the value $1$ ($\frac12$ if the experiment is "fair"). But once I have tossed it and observed the outcome, it's an observation, and that observation doesn't vary, I know what it ...


17

Definition From Wikipedia: A statistic [...] is a single measure of some attribute of a sample (e.g., its arithmetic mean value). And [A]n estimator is a rule for calculating an estimate of a given quantity [of the underlying distribution] based on observed data. The important difference is: A statistic is a function of a sample. An estimator ...


16

If you need a reference for the answer in my comment above, here is one, from Andrew Gelman's blog: Which reminds me of Lucien Le Cam’s reply when I asked him once whether he could think of any examples where the distinction between the strong law of large numbers (convergence with probability 1) and the weak law (convergence in probability) made any ...


16

Let me try it as follows (really not sure if that is useful intuition): Based on my above comment, the correlation will roughly be $$-\frac{E(X)}{\sqrt{E(X^2)}}$$ Thus, if $E(X)>0$ instead of $E(X)=0$, most data will be clustered to the right of zero. Thus, if the slope coefficient gets larger, the correlation formula asserts that the intercept needs to ...


16

You might like to follow Dougherty's Introduction to Econometrics, perhaps considering for now that $x$ is a non-stochastic variable, and defining the mean square deviation of $x$ to be $\DeclareMathOperator{\MSD}{MSD}\MSD(x) = \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2$. Note that the MSD is measured in the square of the units of $x$ (e.g. if $x$ is in $\...


16

A general answer is that an estimator based on a method of moments is not invariant by a bijective change of parameterisation, while a maximum likelihood estimator is invariant. Therefore, they almost never coincide. (Almost never across all possible transforms.) Furthermore, as stated in the question, there are many MoM estimators. An infinity of them, ...


15

This thread is a little old, but it appears that Wikipedia may have changed its definition and if it's accurate, it explains it more clearly for me: An "estimator" or "point estimate" is a statistic (that is, a function of the data) that is used to infer the value of an unknown parameter in a statistical model. So a statistic refers to the data ...


15

What hejseb means is that $\sqrt{n}(\hat\theta-\theta)$ is "bounded in probability", loosely speaking that the probability that $\sqrt{n}(\hat\theta-\theta)$ takes on "extreme" values is "small". Now, $\sqrt{n}$ evidently diverges to infinity. If the product of $\sqrt{n}$ and $(\hat\theta-\theta)$ is bounded, that must mean that $(\hat\theta-\theta)$ goes ...


14

Note that the mean of the Cauchy distribution doesn't exist (so we can't assume it to be 0). However, I assume you mean the center of symmetry of the Cauchy (which is both the mode and the median and various other measures of location). Let's call the center $\mu$ and the scale parameter $\sigma\,$: $$f(x; \mu,\sigma) = \frac{1}{\pi\sigma \left[1 + \left(\...


14

Technically what you're describing when you say that your estimator gets closer to the true value as the sample size grows is (as others have mentioned) consistency, or convergence of statistical estimators. This convergence can either be convergence in probability, which says that $\lim_{n \to \infty} P(|\hat{\theta}_n - \theta| > \epsilon) = 0$ for ...


11

Assume that $g(X_0, \ldots, X_n)$ is an unbiased estimator of $1/\lambda$, that is, $$\sum_{(x_0, \ldots, x_n) \in \mathbb{N}_0^{n+1}} g(x_0, \ldots, x_n) \frac{\lambda^{\sum_{i=0}^n x_i}}{\prod_{i=0}^n x_i!} e^{-(n + 1) \lambda} = \frac{1}{\lambda}, \quad \forall \lambda > 0.$$ Then multiplying by $\lambda e^{(n + 1) \lambda}$ and invoking the MacLaurin ...


11

Why is asymptotic normality important for an estimator? I wouldn't say it's important, really, but when it happens, it can be convenient, and the plain fact is, it happens a lot -- for many popular estimators in commonly used models, it is the case that the distribution of an appropriately standardized estimator will be asymptotically normal. So whether I ...


11

$p$-values are not used for estimating any parameter of interest, but for hypothesis testing. For example, you could be interested in estimating population $\mu$ based on the sample you have, or you could be interested in interval estimate of it, but in hypothesis testing scenario you would rather compare the sample mean $\overline x$ with population mean $\...


11

Estimators are random variables. They exhibit properties that we use to assess their quality, advantages, and disadvantages. So it depends what you mean by "is an estimate of." I can say $\hat{\mu}_0 = 0$ is an estimate of $\mu$, but that doesn't mean it's useful or successful (it uses absolutely no information). The three most popular properties of ...


10

Certainly the residuals are some sort of estimators of $\epsilon$ (to be clear, the definition of the residual is the estimator, the observed residual is an estimate). If the model is correct, then they may sometimes be a fairly good estimate. Indeed $e = y - \hat y = X\beta + \epsilon - X(X'X)^{-1} X'(X\beta + \epsilon) = (I - H)\epsilon$, where $H = X(...


9

[Note: This is my answer to the Dec. 19, 2014, version of the question.] If you operate the change of variable $y=x^2$ in your density $$f_X(x|\alpha,\beta,\sigma)=\frac{1}{\Gamma \left( \alpha \right)\beta^{\alpha}}\exp\left\{{-\frac{x^2}{2\sigma^{2}}\frac{1}{\beta}}\right\}\frac{x^{2\alpha-1}}{2^{\alpha-1}\sigma^{2\alpha}}\mathbb{I}_{{\mathbb{R}}^{+}}(x) $...


9

I am not sure if you confuse consistency and unbiasedness. Consistency: The larger the sample size the smaller the variance of the estimator. Depends on sample size Unbiasedness: The expected value of the estimator equals the true value of the parameters Does not depend on sample size So your sentence if you average a bunch of values of $\hat\theta$, ...


9

There are numerous examples where the MLE has smaller mean square error (MSE) than the best available unbiased estimator (though often there are estimators with smaller MSE still). The "standard" example in normal sampling arises in estimating the variance for example. There are many cases where no unbiased estimator exists, such as the inverse of the rate ...


8

In statistics, bias is clearly a property of the estimator. I share your observation that bias is often incorrectly applied to estimates. Your example seems rather innocent in that regard, because a well-meaning instructor could argue that your students assumed that the error of the estimates is so small that it's OK to equate the estimate with the ...


8

An oracle knows the truth: it knows the true subset and is willing to act on it. The oracle property is that the asymptotic distribution of the estimator is the same as the asymptotic distribution of the MLE on only the true support. That is, the estimator adapts to knowing the true support without paying a price (in terms of the asymptotic distribution.) ...


8

As essentially discussed in the comments, unbiasedness is a finite sample property, and if it held it would be expressed as $$E (\hat \beta ) = \beta$$ (where the expected value is the first moment of the finite-sample distribution) while consistency is an asymptotic property expressed as $$\text{plim} \hat \beta = \beta$$ The OP shows that even though ...


8

I suspect you have solved the maximum likelihood estimation problem incorrectly, most likely by ignoring the parameter constraint. From the stated density, your log-likelihood function is: $$\begin{equation} \begin{aligned} l_\boldsymbol{x}(\theta) &= -n \ln \theta - \sum_{i=1}^n \frac{x_i-\theta}{\theta} \\[8pt] &=-n \ln \theta - \frac{n \bar{x}_n}...


Only top voted, non community-wiki answers of a minimum length are eligible