# What is the difference between an estimator and a statistic?

I learned that a statistic is an attribute you can obtain from samples.Taking many samples of same size, calculating this attribute for all of them and plotting the pdf, we get the distribution of the corresponding attribute or the distribution of the corresponding statistics.

I also heard that statistics are made to be estimators, how do these two concepts differ?

-
Thanks for all the answears ... The concept is a lot more clear to me now.. – gutto Jan 15 '13 at 16:19

From Wikipedia:

A statistic [...] is a single measure of some attribute of a sample (e.g., its arithmetic mean value).

And

[A]n estimator is a rule for calculating an estimate of a given quantity [of the underlying distribution] based on observed data.

The important difference is:

• A statistic relates to a sample.
• An estimator relates to a sample with respect to some quantity.

(For what "Quantity" means, see section below.)

So you can turn a statistic into an estimator by simply spelling out the quantity you want it to estimate; the value doesn't change.

# A statistic is not an estimator

The difference is very subtle and I think it is very common to call a statistic an estimator, e.g. saying sample mean is an estimator for distribution mean. This may be very confusing when you don't properly know both concepts. In particular you cannot calculate the properties of an estimator (e.g. bias) for a mere statistic. Saying "(uncorrected) sample variance is biased" doesn't make much sense, because it is only biased in its function of an estimator for the distribution variance. Thus, formally a statistic cannot be an estimator because you cannot do everything with a statistic which you can do with an estimator (e.g. compute its bias).

Also you can see, that the same statistic can be multiple estimators. E.g. you can estimate both the expected value and the variance of the distribution by, e.g. the sample mean (though sample mean might be a bad estimator for distribution variance). Both 'estimators' will have different properties (e.g. different error, different bias), still underyling them is the same statistic, the same numerical value.

# Quantities of distributions and quantities of samples

Here quantity refers to some property of the distribution, which is usually unknown and thus has to be estimated. This is in contrast to a statistic, which is a property of a sample, e.g. the distribution mean is a quantity of your distribution, while the sample mean is a statistic (a quantity of your sample).

-
There's nothing overtly wrong with these quotations, but they leave me puzzling over what exactly is meant by "quantity." For instance, the quotations do not appear to rule out the possibility that a "quantity" is another statistic based on the same data or perhaps is another statistic based on a separate set of similar data. (In the latter case the first statistic might be used as a predictor. In the former case I don't think there's a name for it, but it definitely is not "estimator.") – whuber Jan 15 '13 at 13:27
@whuber See edit. Initially I wanted to give a short answer... :( – ziggystar Jan 15 '13 at 14:09
+1 Your final paragraph says it all. – whuber Jan 15 '13 at 14:23
Presumably the sample mean and sample median will only estimate the same underlying value if the distribution is one where median=mean... – Stumpy Joe Pete Jan 15 '13 at 16:26
My criticism makes less sense in light of your edit. I was merely saying that in many distributions median != mean, so the sample median and sample mean will not converge to the same value in such cases (i.e., don't estimate the same thing). – Stumpy Joe Pete Jan 17 '13 at 0:59

This thread is a little old, but it appears that Wikipedia may have changed its definition and if it's accurate, it explains it more clearly for me:

An "estimator" or "point estimate" is a statistic (that is, a function of the data) that is used to infer the value of an unknown parameter in a statistical model.

So a statistic refers to the data itself and a calculation with that data. While an estimator refers to a parameter in a model.

If I understand it correctly, then, the mean is a statistic and may also be an estimator. The mean of a sample is a statistic (sum of the sample divided by the sample size). The mean of a sample is also an estimator of the mean of the population, assuming it's normally distributed.

I'd ask @whuber and others who really know this stuff if the (new?) Wikipedia quote is accurate.

-
+1 I think you have it basically right. You might be interested to know that the target of an estimator does not necessarily have to be a particular "parameter" of a model: it can be any property of the model, such as a function of its parameters. For instance, $\mu^2$ is not a parameter for a Normal$(\mu,\sigma^2)$ model, but it can be estimated. – whuber Jun 15 '13 at 16:47

"6" is an example of an estimator. Say your question was, "what is the slope of the best linear function mapping x to y?" Your answer could be "6". Or it could be $(X'X)^{-1}X'Y$. Both are estimators. Which one is better is left to you to decide.

A really good TA once explained the concept of an estimator to me that way.

Basically, an estimator is a thing that you apply to data to get a quantity that you don't know the value of. You know the value of a statistic -- it is a function of the data with no "best" or "optimal" about it. There is no "best" mean. There is just a mean.

Say you have a dataset on number of goats owned per person, and each person's happiness. You're interested in how people's happiness changes with the number of goats they own. An estimator can help you to estimate that relationship from your data. Statistics are just functions of the data that you have. For example, the variance of goat ownership may equal 7. Te forula for calculating variance would be identical between goats and toasters, or whether you're interested in happiness or propensity to get cancer. In that sense, all sensible estimators are statistics.

-

Interesting question. Estimators and statistics do not need to be different things, though. They are different concepts.

A statistic is a function (in broad terms) in which the input is (statistical) data. The effect is that you gain a result, usually a number, from this statistic. In a more abstract term, a statistic may yield more than one number. The statistic depends on the data, but the procedure is deterministic. So the statistic may be: "Sum all numbers and divide by the count" or, in the broader sense "take the gdp data and prepare a report on it".
In the statistical sense we are of course talking about a mathematical function as a statistic.

The significance of this is that if you know properties of the data you input (for example it beeing a random variable), then you can calculate the properties of your statistic, without actually putting in empirical data.

Estimators are estimators because of you intent: to estimate a property. As it turns out, some statistics are good estimators.
For example if you pull data points out of a pool of i.i.d. variables, then the arithmetic mean - a statistic based on the data you pull, will probably be a good estimator for the expected value of that distribution. But then again any thing that produces an estimate is an estimator.

In practice, estimators that you use will be statistics, but there are statistics that aren't estimators. For example test-statistics - though one can argue about the semantics of this statement and to make matters worse, a test statistic may not only be but also include estimators. Though conceptually this doesn't have to be the case.

And of course you can have estimators that aren't statistics, though they are probably not very good at estimating.

-
Could you elaborate a little on that last sentence? For instance, consider an iid sample of size $2n$. I will estimate the median of the population by using a coin flip to choose among the $n$th and $n+1$st largest values in the sample. According to your definition this is not a statistic, because it is not a "deterministic" procedure (although it is a statistic according to a common more general definition). It also is a reasonably good estimator. So I am wondering just what kind of object you have in mind when you refer to an "estimator" that is not a "statistic." – whuber Jan 15 '13 at 13:31
Yeah I would argue that "choosing a value" is the deterministic statistic and everything beforehand is related to modification of the sample you chose from. Then again since the "procedure" if you will - is deterministic I may just allow stochastic elements such this in my definition of statistic... Point beeing that estimators which are not a statistic could be at least those which are independent of any data. For example the number "6" in the answer below. Please note that I did not say that non-statistic estimators are necessarily bad. – IMA Jan 17 '13 at 7:52
I think perhaps you are making too many fine distinctions which are unnecessary and, in the end, complicate your exposition. For example, "1/2" is a great estimator of the parameter of a Bernoulli variable (it is minimax for quadratic loss), so it would be a shame to rule it out just because it is independent of the data. (That would be analogous to ruling out squares as examples of rectangles in Euclidean geometry: you could do that, but that would then double the lengths of most statements concerning properties of rectangles.) It similarly helps not to rule out randomized statistics. – whuber Jan 17 '13 at 15:28
I don't think we are really talking about the same thing. Where do I rule out anything? If one-half is a great estimator, then it is a case where it is. I just don't think that the majority of possible estimators not beeing statistics are pretty great. For a Bernoulli variable "1/2" is good. But -quite- a few other estimators from the class "A real number" are not very good, wouldn't you agree? On the matter of randomized statistics still based on data- I didn't rule it out as I would still say you will require a deterministic procedure. But I concede that I should add this above. – IMA Jan 18 '13 at 8:08

I think a better understanding about what is a sample helps.

[Updated: Sample is a very broad concept, I was talking about "the random sample" . I don't know whether an estimator makes sense or not when the sample is not random.]

from wikipedia:

A random sample is defined as a sample where each individual member of the population has a known, non-zero chance of being selected as part of the sample.

An estimator is a function of a sample. A sample is actually a set of (say, $n$) i.i.d. random variables. That means an estimator is also a function of random variables. An estimator defines a measurement, but not the values of an actual measurement. But we can call it, "the rule for estimating a given quantity based on observed data." Because based on $n$ specific experiments, we can have $n$ specific values for the $n$ i.i.d. random variables. And we get a specific value of the size-$n$ sample.

We replace the sample in the estimator by the value of the sample. We get a value of the estimator, this is a specific measure. And this specific measure is a statistic.

(Check this link for the definition of an estimator, the last sentence reveals why we are always confused.)

-

A good estimator has a sampling distribution that is

1. is centered around the parameter
2. has as small standard error as possible.
-

The Goal of This Piece of Writing:

What I want to do here is to provide you with the similarities and differences between the two intimately related concepts called "statistic" and "estimator". However, I do not want to go through the differences between a parameter and a statistic, which I assume is clear enough to everyone who is struggling with the differences between a statistic and an estimator. If it is not the case for you, you need to study earlier posts first, and then start studying this post.

Relationship:

Basically, any real-valued function of observable random variables in a sample is called a statistic. There are some statistics that if they are well designed, and have some good properties (e.g. consistency, ... ), they can be used to estimate the parameters of the underlying distribution of the population. Therefore, statistics are a large set, and estimators are a subset inside the set of statistics. Hence, every estimator is a statistic, but not every statistic is an estimator.

Similarities:

Speaking of the similarities, as mentioned earlier, both are functions of random variables. In addition, both have distributions called "sampling distributions."

Differences:

Speaking of the differences, they are different in terms of their goals and tasks. The goals and tasks of a statistic could be summarizing the information in a sample (by using sufficient statistics), and sometimes doing hypothesis test, etc. In contrast, the primary goal and task of an estimator, as its name implies, is to estimate the parameters of the population being studied. It is important to mention that there are a wide variety of estimators, each of which has its own computational logic behind, such as MOMEs, MLEs, OLS estimators and so on. Another difference between these two concepts has to do with their desired properties. While one of the most desired properties of a statistic is "sufficiency", the desired properties of an estimator are things like "consistency", "unbiasedness", "precision", etc.

Caution:

Therefore, you need to be careful about using terminology correctly when dealing with statistics and estimators. For instance, it does not make much sense to talk about the biasedness of a mere statistic, which is by no means an estimator, because there is no parameter involved in such a context in order for us to be able to calculate the bias, and talk about it. Thus, you need to be careful about the terminology!

The Bottom Line:

To sum up, any function of observable random variables in a sample is a statistic. If a statistic has capability to estimate a parameter of a population, then we call it an estimator (of the parameter of interest). However, there are some statistics that are not designed to estimate parameters, so these statistics are not estimators, and here we call them "mere statistics".

What I offered above is the way I look at and think of these two concepts, and I tried my best to put it in simple words. I hope it helps!

-

New answer to an old Q:

Definition 1. A statistic is a function that maps each sample to a real number.

Every estimator is a statistic.

But we tend to call only those statistics that are used to generate estimates ("guesses") some parameter an estimator.

So for example, the t-statistic and the sample mean are BOTH statistics. The sample mean is also an estimator (because we often use it to estimate the true population mean).

In contrast, we rarely/never call the t-statistic an estimator, because we rarely/never use it to estimate any parameter.

In the example below, $P$ is a statistic, but not an estimator. While $Q$ is both a statistic and an estimator.



$$\underline{\text{Example}}$$

Suppose our parameter-of-interest is the average outcome $\theta$ of a die-roll.

$\theta$ is some fixed real number that is perhaps known only to God. Nonetheless, we can try to estimate it.

Here's one possible method. We roll a die 3 times.

A sample is any $\textbf{s}=\left(x_1,x_2,x_3\right)$, where $x_1$ is the outcome of the first roll, $x_2$ that of the second, and $x_3$ that of the third.

Here are three examples of samples: $\textbf{s}_1=\left(5,4,1\right)$, $\textbf{s}_2=\left(4,1,6\right)$, and $\textbf{s}_3=\left(6,3,2\right)$.

Here are two examples of statistics $P$ and $Q$ (remember that a statistic is simply a function). Define $P$ and $Q$ by: For any $\textbf{s}=\left(x_1,x_2,x_3\right)$, $$P(\textbf{s})=\frac{x_1}{\ln(x_2+x_3)},$$ $$Q(\textbf{s})=\frac{x_1+x_2+x_3}{3}.$$

The statistic $P$ is a rather-bizarre statistic and is probably not very useful for anything. Nonetheless, it is a statistic all the same, simply because it satisfies the definition of a statistic (it is a function that maps each sample to a real number).

$Q$ is also a statistic. But in addition, it is also an estimator for the parameter $\theta$.

(We could, of course, claim that $P$ is also an estimator for $\theta$. But it would be a very poor estimator that no one would want to use.)

-
This answer is headed in a good direction. "Definition 2," though, does not appear to be a valid definition, because of its circularity (it defines "estimator" in terms of "estimate" without explaining the latter). For it to be effective you need to explain what an "estimate of a parameter" is in sufficient detail and clarity that people can formulate quantitative measurements of how well an estimator works. – whuber Jun 21 at 13:27
@whuber: I'm trying to keep it simple. A parameter is any real number (e.g. the average outcome $\theta$ of a die roll). Informally, an estimate for a parameter is simply a "guess" of what a parameter is. An estimate is thus simply also a real number. (E.g., an estimate of $\theta$ is $5$.) // The question of "how to formulate quantitative measurements of how well an estimator works" is entirely distinct from the simpler and more basic question of the distinction between a statistic and an estimator. Which is the question here. – Kenny LJ Jun 22 at 12:16
Unfortunately, as I was trying to suggest, something essential seems to have been lost in the simplification, because your second definition does not distinguish an estimator from any other statistic at all. – whuber Jun 22 at 13:32
@whuber: That's right. Formally, an estimator is simply a statistic. But we tend to use the word "estimator" to refer to a statistic if that statistic is used to estimate some parameter-of-interest. I have edited my answer to clarify this point. – Kenny LJ Jun 23 at 2:25

In hypothesis testing :

A test-statistic is about hypothesis testing. A test-statistic is a random variable given/under the null hypothesis. Now, some may call a statistic the value/measure of the test-statistic given the sample.

With these two you can get the p-value which is a measure that helps to reject or not reject the null hypothesis. All in all, a statistic is an estimation of how far/close to your hypothesis.