21
$\begingroup$

What is the relation between estimator and estimate?

$\endgroup$
  • 5
    $\begingroup$ "In statistics, an estimator is a rule for calculating an estimate of a given quantity based on observed data: thus the rule and its result (the estimate) are distinguished." (First line of the Wikipedia article en.wikipedia.org/wiki/Estimator ). $\endgroup$ – whuber Feb 24 '11 at 19:15
  • $\begingroup$ +1 I am upvoting this question (despite the presence of a well formulated answer on an obvious Wikipedia page) because initial attempts at answering it here have pointed to some subtleties. $\endgroup$ – whuber Feb 24 '11 at 21:38
  • $\begingroup$ @whuber, can I say the model parameters estimates are the estimator? $\endgroup$ – avocado Dec 26 '13 at 7:16
  • 2
    $\begingroup$ @loganecolss An estimator is a mathematical function. That is distinguished from the value (the estimate) it might attain for any set of data. One way to appreciate the difference is to note that certain sets of data will produce the same estimates of, say, the slope in a linear regression using different estimators (such as Maximum Likelihood or Iteratively Reweighted Least Squares, for instance). Without distinguishing estimates from the estimators used to produce those estimates, we would be unable to understand what that statement even says. $\endgroup$ – whuber Dec 26 '13 at 14:09
  • $\begingroup$ @whuber, even with one certain data set $D$, different estimator could also give different estimates, don't they? $\endgroup$ – avocado Dec 26 '13 at 14:16
13
$\begingroup$

E. L. Lehmann, in his classic Theory of Point Estimation, answers this question on pp 1-2.

The observations are now postulated to be the values taken on by random variables which are assumed to follow a joint probability distribution, $P$, belonging to some known class...

...let us now specialize to point estimation...suppose that $g$ is a real-valued function defined [on the stipulated class of distributions] and that we would like to know the value of $g$ [at whatever is the actual distribution in effect, $\theta$]. Unfortunately, $\theta$, and hence $g(\theta)$, is unknown. However, the data can be used to obtain an estimate of $g(\theta)$, a value that one hopes will be close to $g(\theta)$.

In words: an estimator is a definite mathematical procedure that comes up with a number (the estimate) for any possible set of data that a particular problem could produce. That number is intended to represent some definite numerical property ($g(\theta)$) of the data-generation process; we might call this the "estimand."

The estimator itself is not a random variable: it's just a mathematical function. However, the estimate it produces is based on data which themselves are modeled as random variables. This makes the estimate (thought of as depending on the data) into a random variable and a particular estimate for a particular set of data becomes a realization of that random variable.

In one (conventional) ordinary least squares formulation, the data consist of ordered pairs $(x_i, y_i)$. The $x_i$ have been determined by the experimenter (they can be amounts of a drug administered, for example). Each $y_i$ (a response to the drug, for instance) is assumed to come from a probability distribution that is Normal but with unknown mean $\mu_i$ and common variance $\sigma^2$. Furthermore, it is assumed that the means are related to the $x_i$ via a formula $\mu_i = \beta_0 + \beta_1 x_i$. These three parameters--$\sigma$, $\beta_0$, and $\beta_1$--determine the underlying distribution of $y_i$ for any value of $x_i$. Therefore any property of that distribution can be thought of as a function of $(\sigma, \beta_0, \beta_1)$. Examples of such properties are the intercept $\beta_0$, the slope $\beta_1$, the value of $\cos(\sigma + \beta_0^2 - \beta_1)$, or even the mean at the value $x=2$, which (according to this formulation) must be $\beta_0 + 2 \beta_1$.

In this OLS context, a non-example of an estimator would be a procedure to guess at the value of $y$ if $x$ were set equal to 2. This is not an estimator because this value of $y$ is random (in a way completely separate from the randomness of the data): it is not a (definite numerical) property of the distribution, even though it is related to that distribution. (As we just saw, though, the expectation of $y$ for $x=2$, equal to $\beta_0 + 2 \beta_1$, can be estimated.)

In Lehmann's formulation, almost any formula can be an estimator of almost any property. There is no inherent mathematical link between an estimator and an estimand. However, we can assess--in advance--the chance that an estimator will be reasonably close to the quantity it is intended to estimate. Ways to do this, and how to exploit them, are the subject of estimation theory.

$\endgroup$
  • 1
    $\begingroup$ (+1) A very precise and detailed response. $\endgroup$ – chl Feb 25 '11 at 9:14
  • 2
    $\begingroup$ Is not a function of a random variable itself also a random variable? $\endgroup$ – jsk Mar 19 '15 at 19:18
  • $\begingroup$ @jsk I think the distinction I was trying to make here may be clarified by considering the composition of functions $$\Omega\to\mathbb{R}^n\to\mathbb{R}.$$ The first function is a random variable $X$; the second one (call it $t$) is termed an estimator here, and the composition of the two $$t\circ X:\Omega\to\mathbb{R}$$ is an "estimate" or "estimation procedure," which is--as you correctly say--a random variable. $\endgroup$ – whuber Mar 19 '15 at 20:03
  • 1
    $\begingroup$ @whuber In your post, you say "The estimator itself is not a random variable." I attempted an edit to your post to clarify the point that you and I seem to agree on, but it seems someone rejected my edit. Perhaps they would prefer your edit! $\endgroup$ – jsk Mar 19 '15 at 23:19
  • $\begingroup$ Let us continue this discussion in chat. $\endgroup$ – whuber Nov 20 '18 at 15:42
7
$\begingroup$

In short: an estimator is a function and an estimate is a value that summarizes an observed sample.

An estimator is a function that maps a random sample to the parameter estimate:

$$ \hat{\Theta}=t(X_1,X_2,...,X_n) $$ Note that an estimator of n random variables $X_1,X_2,...,X_n$ is a random variable $\hat{\Theta}$. For instance, an estimator is the sample mean: $$ \overline{X}=\frac{1}{n}\sum_{n=1}^nX_i $$ An estimate $\hat{\theta}$ is the result of applying the estimator function to a lowercase observed sample $x_1,x_2,...,x_n$:

$$ \hat{\theta}=t(x_1,x_2,...,x_n) $$ For instance, an estimate of the observed sample $x_1,x_2,...,x_n$ is the sample mean: $$ \hat{\mu}=\overline{x}=\frac{1}{n}\sum_{n=1}^nx_i $$

$\endgroup$
  • $\begingroup$ estimator is a RV, while estimate is a constant? $\endgroup$ – Parthiban Rajendran Nov 18 '18 at 15:13
  • $\begingroup$ Isn't your conclusion conflicting with @whuber's? Here you say estimator is RV, but whuber says otherwise. $\endgroup$ – Parthiban Rajendran Nov 20 '18 at 4:03
  • $\begingroup$ Yes, I disagree with @whuber's statement "The estimator itself is not a random variable: it's just a mathematical function". A function of random variable is also a random variable. onlinecourses.science.psu.edu/stat414/node/128 $\endgroup$ – Freeman Nov 27 '18 at 16:35
3
$\begingroup$

It might be helpful to illustrate whuber's answer in the context of a linear regression model. Let's say you have some bivariate data and you use Ordinary Least Squares to come up with the following model:

Y = 6X + 1

At this point, you can take any value of X, plug it into the model and predict the outcome, Y. In this sense, you might think of the individual components of the generic form of the model (mX + B) as estimators. The sample data (which you presumably plugged into the generic model to calculate the specific values for m and B above) provided a basis on which you could come up with estimates for m and B respectively.

Consistent with @whuber's points in our thread below, whatever values of Y a particular set of estimators generate you for are, in the context of linear regression, thought of as predicted values.

(edited -- a few times -- to reflect the comments below)

$\endgroup$
  • 1
    $\begingroup$ You have nicely defined a predictor. It is subtly (but importantly) different from an estimator. The estimator in this context is the least squares formula used to compute the parameters 1 and 6 from the data. $\endgroup$ – whuber Feb 24 '11 at 21:36
  • $\begingroup$ Hmm, I didn't mean it that way, @whuber, but I think your comment illustrates an important ambiguity in my language that I didn't notice before. The main point here is that you can think of the generic form of the equation Y = mX + B (as used above) as an estimator, whereas the particular predicted values generated by specific examples of that formula (e.g., 1 + 6X) are estimates. Let me try to edit the paragraph above to capture that distinction... $\endgroup$ – ashaw Feb 24 '11 at 21:53
  • $\begingroup$ btw, I'm trying to explain this without introducing the "hat" notation that I've encountered in most textbook discussions of this concept. Perhaps that's the better route after all? $\endgroup$ – ashaw Feb 24 '11 at 21:58
  • 2
    $\begingroup$ I think you have hit a nice medium between accuracy and technicality in your original answer: keep it up! You don't need hats, but if you can manage to show how an estimator is distinguished from other, similar-looking things, that would be most helpful. But please notice the distinction between predicting a value Y and estimating a parameter such as m or b. Y could be interpreted as a random variable; m and b are not (except in a Bayesian setting). $\endgroup$ – whuber Feb 24 '11 at 22:15
  • $\begingroup$ indeed, a very good point in terms of parameters versus values there. Editing again... $\endgroup$ – ashaw Feb 24 '11 at 22:24
0
$\begingroup$

Suppose you received some data, and you had some observed variable called theta. Now your data can be from a distribution of data, for this distribution, there is a corresponding value of theta that you infer which is a random variable. You can use the MAP or mean for calculating the estimate of this random variable whenever the distribution of your data changes. So the random variable theta is known as an estimate, a single value of the unobserved variable for a particular type of data.

While estimator is your data, which is also a random variable. For different types of distributions you have different types of data and thus you have a different estimate and thus this corresponding random variable is called the estimator.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy