# What is the relation between estimator and estimate?

What is the relation between estimator and estimate?

• "In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule and its result (the estimate) are distinguished." (First line of the Wikipedia article en.wikipedia.org/wiki/Estimator ).
– whuber
Commented Feb 24, 2011 at 19:15
• +1 I am upvoting this question (despite the presence of a well formulated answer on an obvious Wikipedia page) because initial attempts at answering it here have pointed to some subtleties.
– whuber
Commented Feb 24, 2011 at 21:38
• @whuber, can I say the model parameters estimates are the estimator? Commented Dec 26, 2013 at 7:16
• @loganecolss An estimator is a mathematical function. That is distinguished from the value (the estimate) it might attain for any set of data. One way to appreciate the difference is to note that certain sets of data will produce the same estimates of, say, the slope in a linear regression using different estimators (such as Maximum Likelihood or Iteratively Reweighted Least Squares, for instance). Without distinguishing estimates from the estimators used to produce those estimates, we would be unable to understand what that statement even says.
– whuber
Commented Dec 26, 2013 at 14:09
• Great work Bill Huber. Your comments is a very simple but clear and correct. Nothing else need be said other than correcting others who still don't understand it. Commented Jan 2, 2017 at 7:02

E. L. Lehmann, in his classic Theory of Point Estimation, answers this question on pp 1-2.

The observations are now postulated to be the values taken on by random variables which are assumed to follow a joint probability distribution, $$P$$, belonging to some known class...

...let us now specialize to point estimation...suppose that $$g$$ is a real-valued function defined [on the stipulated class of distributions] and that we would like to know the value of $$g$$ [at whatever is the actual distribution in effect, $$\theta$$]. Unfortunately, $$\theta$$, and hence $$g(\theta)$$, is unknown. However, the data can be used to obtain an estimate of $$g(\theta)$$, a value that one hopes will be close to $$g(\theta)$$.

In words: an estimator is a definite mathematical procedure that comes up with a number (the estimate) for any possible set of data that a particular problem could produce. That number is intended to represent some definite numerical property ($$g(\theta)$$) of the data-generation process; we might call this the "estimand."

The estimator itself is not a random variable: it's just a mathematical function. However, the estimate it produces is based on data which themselves are modeled as random variables. This makes the estimate (thought of as depending on the data) into a random variable and a particular estimate for a particular set of data becomes a realization of that random variable.

In one (conventional) ordinary least squares formulation, the data consist of ordered pairs $$(x_i, y_i)$$. The $$x_i$$ have been determined by the experimenter (they can be amounts of a drug administered, for example). Each $$y_i$$ (a response to the drug, for instance) is assumed to come from a probability distribution that is Normal but with unknown mean $$\mu_i$$ and common variance $$\sigma^2$$. Furthermore, it is assumed that the means are related to the $$x_i$$ via a formula $$\mu_i = \beta_0 + \beta_1 x_i$$. These three parameters--$$\sigma$$, $$\beta_0$$, and $$\beta_1$$--determine the underlying distribution of $$y_i$$ for any value of $$x_i$$. Therefore any property of that distribution can be thought of as a function of $$(\sigma, \beta_0, \beta_1)$$. Examples of such properties are the intercept $$\beta_0$$, the slope $$\beta_1$$, the value of $$\cos(\sigma + \beta_0^2 - \beta_1)$$, or even the mean at the value $$x=2$$, which (according to this formulation) must be $$\beta_0 + 2 \beta_1$$.

In this OLS context, a non-example of an estimator would be a procedure to guess at the value of $$y$$ if $$x$$ were set equal to 2. This is not an estimator because this value of $$y$$ is random (in a way completely separate from the randomness of the data): it is not a (definite numerical) property of the distribution, even though it is related to that distribution. (As we just saw, though, the expectation of $$y$$ for $$x=2$$, equal to $$\beta_0 + 2 \beta_1$$, can be estimated.)

In Lehmann's formulation, almost any formula can be an estimator of almost any property. There is no inherent mathematical link between an estimator and an estimand. However, we can assess--in advance--the chance that an estimator will be reasonably close to the quantity it is intended to estimate. Ways to do this, and how to exploit them, are the subject of estimation theory.

• (+1) A very precise and detailed response.
– chl
Commented Feb 25, 2011 at 9:14
• Is not a function of a random variable itself also a random variable?
– jsk
Commented Mar 19, 2015 at 19:18
• @jsk I think the distinction I was trying to make here may be clarified by considering the composition of functions $$\Omega\to\mathbb{R}^n\to\mathbb{R}.$$ The first function is a random variable $X$; the second one (call it $t$) is termed an estimator here, and the composition of the two $$t\circ X:\Omega\to\mathbb{R}$$ is an "estimate" or "estimation procedure," which is--as you correctly say--a random variable.
– whuber
Commented Mar 19, 2015 at 20:03
• @whuber In your post, you say "The estimator itself is not a random variable." I attempted an edit to your post to clarify the point that you and I seem to agree on, but it seems someone rejected my edit. Perhaps they would prefer your edit!
– jsk
Commented Mar 19, 2015 at 23:19
• – whuber
Commented Nov 20, 2018 at 15:42

In short: an estimator is a function and an estimate is a value that summarizes an observed sample.

An estimator is a function that maps a random sample to the parameter estimate:

$$\hat{\Theta}=t(X_1,X_2,...,X_n)$$ Note that an estimator of n random variables $X_1,X_2,...,X_n$ is a random variable $\hat{\Theta}$. For instance, an estimator is the sample mean: $$\overline{X}=\frac{1}{n}\sum_{n=1}^nX_i$$ An estimate $\hat{\theta}$ is the result of applying the estimator function to a lowercase observed sample $x_1,x_2,...,x_n$:

$$\hat{\theta}=t(x_1,x_2,...,x_n)$$ For instance, an estimate of the observed sample $x_1,x_2,...,x_n$ is the sample mean: $$\hat{\mu}=\overline{x}=\frac{1}{n}\sum_{n=1}^nx_i$$

• estimator is a RV, while estimate is a constant? Commented Nov 18, 2018 at 15:13
• Isn't your conclusion conflicting with @whuber's? Here you say estimator is RV, but whuber says otherwise. Commented Nov 20, 2018 at 4:03
• Yes, I disagree with @whuber's statement "The estimator itself is not a random variable: it's just a mathematical function". A function of random variable is also a random variable. onlinecourses.science.psu.edu/stat414/node/128 Commented Nov 27, 2018 at 16:35

It might be helpful to illustrate whuber's answer in the context of a linear regression model. Let's say you have some bivariate data and you use Ordinary Least Squares to come up with the following model:

Y = 6X + 1

At this point, you can take any value of X, plug it into the model and predict the outcome, Y. In this sense, you might think of the individual components of the generic form of the model (mX + B) as estimators. The sample data (which you presumably plugged into the generic model to calculate the specific values for m and B above) provided a basis on which you could come up with estimates for m and B respectively.

Consistent with @whuber's points in our thread below, whatever values of Y a particular set of estimators generate you for are, in the context of linear regression, thought of as predicted values.

(edited -- a few times -- to reflect the comments below)

• You have nicely defined a predictor. It is subtly (but importantly) different from an estimator. The estimator in this context is the least squares formula used to compute the parameters 1 and 6 from the data.
– whuber
Commented Feb 24, 2011 at 21:36
• Hmm, I didn't mean it that way, @whuber, but I think your comment illustrates an important ambiguity in my language that I didn't notice before. The main point here is that you can think of the generic form of the equation Y = mX + B (as used above) as an estimator, whereas the particular predicted values generated by specific examples of that formula (e.g., 1 + 6X) are estimates. Let me try to edit the paragraph above to capture that distinction... Commented Feb 24, 2011 at 21:53
• btw, I'm trying to explain this without introducing the "hat" notation that I've encountered in most textbook discussions of this concept. Perhaps that's the better route after all? Commented Feb 24, 2011 at 21:58
• I think you have hit a nice medium between accuracy and technicality in your original answer: keep it up! You don't need hats, but if you can manage to show how an estimator is distinguished from other, similar-looking things, that would be most helpful. But please notice the distinction between predicting a value Y and estimating a parameter such as m or b. Y could be interpreted as a random variable; m and b are not (except in a Bayesian setting).
– whuber
Commented Feb 24, 2011 at 22:15
• indeed, a very good point in terms of parameters versus values there. Editing again... Commented Feb 24, 2011 at 22:24

Let $$\left(\mathcal X, \mathcal A, \left(\mathbb P_\vartheta\right)_{\vartheta \in \Theta}\right)$$ be a statistical model consisting of

• a set $$\mathcal X$$ (the sample space),
• a $$\sigma$$-algebra $$\mathcal A$$ on $$\mathcal X$$,
• a family of probability measures $$\left(\mathbb P_\vartheta\right)_{\vartheta \in \Theta}$$ on $$\mathcal A$$ with
• index set $$\Theta$$ (the parameter space) of cardinality bigger than one.

Let $$(\Sigma, \mathcal S)$$ be another event space, i.e., a measurable space.

Let $$\tau: \Theta \to \Sigma$$ be any function, whose value at $$\vartheta \in \Theta$$ is to be estimated.

Then, an estimator $$T$$ of (the estimand) $$\tau$$ is defined to be any random variable from $$(\mathcal X, \mathcal A)$$ to $$(\Sigma, \mathcal S)$$. One also says that $$T$$ is an estimator of (the estimand) $$\tau(\vartheta), \vartheta \in \Theta$$.
For any $$x \in \mathcal X$$, the realization $$T(x)$$ is called the corresponding estimate.

Clearly, we want an estimate to be "close" to its estimand at any true (but unknown) parameter value $$\vartheta \in \Theta$$. But in the definitions this is not formalized beyond the common codomain of estimands and their estimators to avoid unnecessary narrowing of the concepts.

Unfortunately, some people use the term estimate for both estimators and estimates (as defined above). For example, Cox and Hinkley write the following in Theoretical Statistics on page 252:

Suppose that we have a scalar parameter $$\theta$$, and let $$T$$ be a statistic and $$t$$ its observed value. When estimating $$\theta$$, $$T$$ is sometimes called an estimator, $$t$$ an estimate; we shall use the single term estimate.

References

• Cox, D. R., & Hinkley, D. V. (1974). Theoretical statistics. Chapman & Hall.
• Georgii, H.-O. (2013). Stochastics: Introduction to probability and statistics (E. Baake & M. Ortgiese, Trans.). Walter de Gruyter.
• Lehmann, E. L., & Casella, G. (1998). Theory of point estimation. Springer.
• Just a small precision, am I right to think that in fact, $\tau$ could just be the identity in the simplest case ? I mean, in the case where we are just looking for an estimate/estimator of $\vartheta$ itself ? Commented May 12 at 8:23
• @MysteryGuy Yes, $\tau = \operatorname{id}_{\Theta}$ is a common special case and (for $\vartheta \in \Theta$) one typically speaks of an estimator of $\vartheta$ (i.e., of $\tau(\vartheta)$) instead of an estimator of $\operatorname{id}_{\Theta}$ (i.e., of $\tau$) in this case. This common terminology is now included in my updated answer. Commented May 12 at 13:39

Suppose you received some data, and you had some observed variable called theta. Now your data can be from a distribution of data, for this distribution, there is a corresponding value of theta that you infer which is a random variable. You can use the MAP or mean for calculating the estimate of this random variable whenever the distribution of your data changes. So the random variable theta is known as an estimate, a single value of the unobserved variable for a particular type of data.

While estimator is your data, which is also a random variable. For different types of distributions you have different types of data and thus you have a different estimate and thus this corresponding random variable is called the estimator.