12
$\begingroup$

I'm relatively familiar with the distinction between the terms statistic and parameter. I see a statistic as the value obtained from applying a function to the sample data. However, most examples of parameters relate to defining a parametric distribution. A common example is the mean and standard deviation to parameterise the normal distribution or the coefficients and error variance to parameterise a linear regression.

However, there are many other values of the population distribution that are less prototypical (e.g., minimum, maximum, r-square in multiple regression, the .25 quantile, median, the number of predictors with non-zero coefficients, skewness, the number of correlations in a correlation matrix greater than .3, etc.).

Thus, my questions are:

  • Should any quantitative property of a population be labelled a "parameter"?
  • If yes, then why?
  • If no, what characteristics should not be labelled a parameter? What should they be labelled? And why?

Elaboration on confusion

The Wikipedia article on estimators states:

An "estimator" or "point estimate" is a statistic (that is, a function of the data) that is used to infer the value of an unknown parameter in a statistical model.

But I can define the unknown value as .25 quantile and I can develop an estimator for that unknown. I.e., not all quantitative properties of a population are parameters in the same way that say the mean and sd are parameters of a normal distribution, yet it is legitimate to seek to estimate any quantitative population property.

$\endgroup$
14
$\begingroup$

This question goes to the heart of what statistics is and how to to conduct a good statistical analysis. It raises many issues, some of terminology and others of theory. To clarify them, let's begin by noting the implicit context of the question and go on from there to define the key terms "parameter," "property," and "estimator." The several parts of the question are answered as they come up in the discussion. The final concluding section summarizes the key ideas.

State spaces

A common statistical use of "the distribution," as in "the Normal distribution with PDF proportional to $\exp(-\frac{1}{2}(x-\mu)/\sigma)^2)dx$" is actually a (serious) abuse of English, because obviously this is not one distribution: it's a whole family of distributions parameterized by the symbols $\mu$ and $\sigma$. A standard notation for this is the "state space" $\Omega$, a set of distributions. (I am simplifying a bit here for the sake of exposition and will continue to simplify as we go along, while remaining as rigorous as possible.) Its role is to delineate the possible targets of our statistical procedures: when we estimate something, we are picking out one (or sometimes more) elements of $\Omega$.

Sometimes state spaces are explicitly parameterized, as in $\Omega = \{\mathcal{N}(\mu, \sigma^2)|\mu \in \mathbb{R}, \sigma \gt 0\}$. In this description there is a one-to-one correspondence between the set of tuples $\{(\mu,\sigma)\}$ in the upper half plane and the set of distributions we will be using to model our data. One value of such a parameterization is that we may now refer concretely to distributions in $\Omega$ by means of an ordered pair of real numbers.

In other cases state spaces are not explicitly parameterized. An example would be the set of all unimodal continuous distributions. Below, we will address the question of whether an adequate parameterization can be found in such cases anyway.

Parameterizations

Generally, a parameterization of $\Omega$ is a correspondence (mathematical function) from a subset of $\mathbb{R}^d$ (with $d$ finite) to $\Omega$. That is, it uses ordered sets of $d$-tuples to label the distributions. But it's not just any correspondence: it has to be "well behaved." To understand this, consider the set of all continuous distributions whose PDFs have finite expectations. This would widely be regarded as "non-parametric" in the sense that any "natural" attempt to parameterize this set would involve a countable sequence of real numbers (using an expansion in any orthogonal basis). Nevertheless, because this set has cardinality $\aleph_1$, which is the cardinality of the reals, there must exist some one-to-one correspondence between these distributions and $\mathbb{R}$. Paradoxically, that would seem to make this a parameterized state space with a single real parameter!

The paradox is resolved by noting that a single real number cannot enjoy a "nice" relationship with the distributions: when we change the value of that number, the distribution it corresponds to must in some cases change in radical ways. We rule out such "pathological" parameterizations by requiring that distributions corresponding to close values of their parameters must themselves be "close" to one another. Discussing suitable definitions of "close" would take us too far afield, but I hope this description is enough to demonstrate that there is much more to being a parameter than just naming a particular distribution.

Properties of distributions

Through repeated application, we become accustomed to thinking of a "property" of a distribution as some intelligible quantity that frequently appears in our work, such as its expectation, variance, and so on. The problem with this as a possible definition of "property" is that it's too vague and not sufficiently general. (This is where mathematics was in the mid-18th century, where "functions" were thought of as finite processes applied to objects.) Instead, about the only sensible definition of "property" that will always work is to think of a property as being a number that is uniquely assigned to every distribution in $\Omega$. This includes the mean, the variance, any moment, any algebraic combination of moments, any quantile, and plenty more, including things that cannot even be computed. However, it does not include things that would make no sense for some of the elements of $\Omega$. For instance, if $\Omega$ consists of all Student t distributions, then the mean is not a valid property for $\Omega$ (because $t_1$ has no mean). This impresses on us once again how much our ideas depend on what $\Omega$ really consists of.

Properties are not always parameters

A property can be such a complicated function that it would not serve as a parameter. Consider the case of the "Normal distribution." We might want to know whether the true distribution's mean, when rounded to the nearest integer, is even. That's a property. But it will not serve as a parameter.

Parameters are not necessarily properties

When parameters and distributions are in one-to-one correspondence then obviously any parameter, and any function of the parameters for that matter, is a property according to our definition. But there need not be a one-to-one correspondence between parameters and distributions: sometimes a few distributions must be described by two or more distinctly different values of the parameters. For instance, a location parameter for points on the sphere would naturally use latitude and longitude. That's fine--except at the two poles, which correspond to a given latitude and any valid longitude. The location (point on the sphere) indeed is a property but its longitude is not necessarily a property. Although there are various dodges (just declare the longitude of a pole to be zero, for instance), this issue highlights the important conceptual difference between a property (which is uniquely associated with a distribution) and a parameter (which is a way of labeling the distribution and might not be unique).

Statistical procedures

The target of an estimate is called an estimand. It is merely a property. The statistician is not free to select the estimand: that is the province of her client. When someone comes to you with a sample of a population and asks for you to estimate the population's 99th percentile, you would likely be remiss in supplying an estimator of the mean instead! Your job, as statistician, is to identify a good procedure for estimating the estimand you have been given. (Sometimes your job is to persuade your client that he has selected the wrong estimand for his scientific objectives, but that's a different issue...)

By definition, a procedure is a way to get a number out of the data. Procedures are usually given as formulas to be applied to the data, like "add them all up and divide by their count." Literally any procedure may be pronounced an "estimator" of a given estimand. For instance, I could declare that the sample mean (a formula applied to the data) estimates the population variance (a property of the population, assuming our client has restricted the set of possible populations $\Omega$ to include only those that actually have variances).

Estimators

An estimator needn't have any obvious connection to the estimand. For instance, do you see any connection between the sample mean and a population variance? Neither do I. But nevertheless, the sample mean actually is a decent estimator of the population variance for certain $\Omega$ (such as the set of all Poisson distributions). Herein lies one key to understanding estimators: their qualities depend on the set of possible states $\Omega$. But that's only part of it.

A competent statistician will want to know how well the procedure they are recommending will actually perform. Let's call the procedure "$t$" and let the estimand be $\theta$. Not knowing which distribution actually is the true one, she will contemplate the procedure's performance for every possible distribution $F \in \Omega$. Given such an $F$, and given any possible outcome $s$ (that is, a set of data), she will compare $t(s)$ (what her procedure estimates) to $\theta(F)$ (the value of the estimand for $F$). It is her client's responsibility to tell her how close or far apart those two are. (This is often done with a "loss" function.) She can then contemplate the expectation of the distance between $t(s)$ and $\theta(F)$. This is the risk of her procedure. Because it depends on $F$, the risk is a function defined on $\Omega$.

(Good) statisticians recommend procedures based on comparing risk. For instance, suppose that for every $F \in \Omega$, the risk of procedure $t_1$ is less than or equal to the risk of $t$. Then there is no reason ever to use $t$: it is "inadmissible." Otherwise it is "admissible".

(A "Bayesian" statistician will always compare risks by averaging over a "prior" distribution of possible states (usually supplied by the client). A "Frequentist" statistician might do this, if such a prior justifiably exists, but is also willing to compare risks in other ways Bayesians eschew.)

Conclusions

We have a right to say that any $t$ that is admissible for $\theta$ is an estimator of $\theta$. We must, for practical purposes (because admissible procedures can be hard to find), bend this to saying that any $t$ that has acceptably small risk (when being compared to $\theta$) among practicable procedures is an estimator of $\theta$. "Acceptably" and "practicable" are determined by the client, of course: "acceptably" refers to their risk and "practicable" reflects the cost (ultimately paid by them) of implementing the procedure.

Underlying this concise definition are all the ideas just discussed: to understand it we must have in mind a specific $\Omega$ (which is a model of the problem, process, or population under study), a definite estimand (supplied by the client), a specific loss function (which quantitatively connects $t$ to the estimand and is also given by the client), the idea of risk (computed by the statistician), some procedure for comparing risk functions (the responsibility of the statistician in consultation with the client), and a sense of what procedures actually can be carried out (the "practicability" issue), even though none of these are explicitly mentioned in the definition.

$\endgroup$
  • 2
    $\begingroup$ @Nick Cox, in his reply, brings up some excellent points that (in my interpretation) go to "what do we do when we know that any model $\Omega$ and any loss function we specify are going to be somewhat inaccurate or inadequate?" The answer to that would take us in a different direction; all I want to say here is that the framework I have set forth --which is the classical one that Tukey was reacting to--gives us a good basis for thinking about such broader questions of data analysis. At a minimum, it clarifies the implicit assumptions that go into standard terms like "estimator." $\endgroup$ – whuber Jul 5 '13 at 15:55
10
$\begingroup$

As with many questions on definitions, answers need to have an eye both on the underlying principles and on the ways terms are used in practice, which can often be at least a little loose or inconsistent, even by individuals who are well informed, and more importantly, variable from community to community.

One common principle is that a statistic is a property of a sample, and a known constant, and a parameter is the corresponding property of the population, and so an unknown constant. The word "corresponding" is to be understood as quite elastic here. Incidentally, precisely this distinction and precisely this terminology are less than a century old, having being introduced by R.A. Fisher.

But

  1. A set-up of sample and population doesn't characterise all our own problems. Time series are one major class of examples in which the idea is rather of an underlying generating process, and something like that is arguably the deeper and more general idea.

  2. There are set-ups in which parameters change. Again, time series analysis provides examples.

  3. To the main point here, we don't in practice think of all the properties of a population or process as parameters. If some procedure assumes a model of a normal distribution, then the minimum and maximum are not parameters. (Indeed, according to the model, the minimum and maximum are arbitrarily large negative and positive numbers any way, not that that should worry us.)

I would say that for once Wikipedia is pointing in the right direction here, and practice and principle are both respected if we say that a parameter is whatever we are estimating.

This helps too with other questions that have caused puzzlement. For example, if we calculate a 25% trimmed mean, what we are estimating? A reasonable answer is the corresponding property of the population, which in effect is defined by the estimation method. One terminology is that an estimator has an estimand, whatever it is estimating. Starting with some Platonic idea of a property "out there" (say the mode of a distribution) and thinking how to estimate that is reasonable, as is thinking up good recipes for analysing data and thinking through what they imply when regarded as inference.

As often in applied mathematics or science, there is a twofold aspect to a parameter. We often think of it as something real out there which we are discovering, but it is also true that it is something defined by our model of the process, so that it has no meaning outside the context of the model.

Two quite different points:

  1. Many scientists use the word "parameter" in the way that statisticians use variable. I have a scientist persona as well as a statistical one, and I would say that is unfortunate. Variables and properties are better words.

  2. It is remarkably common in wider English usage that parameter is thought to mean limits or bounds, which may stem from some original confusion between "parameter" and "perimeter".

A note on the estimand point of view

The classical position is that we identify a parameter in advance and then decide how to estimate it, and this remains majority practice, but reversing the process is not absurd and can be helpful for some problems. I call this the estimand point of view. It has been in the literature for at least 50 years. Tukey (1962, p.60) urged that

"We must give even more attention to starting with an estimator and discovering what is a reasonable estimand, to discovering what is it reasonable to think of the estimator as estimating."

A similar point of view has been elaborated formally in considerable detail and depth by Bickel and Lehmann (1975) and informally with considerable lucidity by Mosteller and Tukey (1977, pp.32-34).

There is also an elementary version. Using (say) sample median or geometric mean to estimate the corresponding population parameter makes sense regardless of whether the underlying distribution is symmetric, and the same goodwill can be extended to (e.g.) sample trimmed means, which are regarded as estimators of their population counterparts.

Bickel, P.J. and E.L. Lehmann. 1975. Descriptive statistics for nonparametric models. II. Location. Annals of Statistics 3: 1045-1069.

Mosteller, F. and J.W. Tukey. 1977. Data Analysis and Regression. Reading, MA: Addison-Wesley.

Tukey, J.W. 1962. The future of data analysis. Annals of Mathematical Statistics 33: 1-67.

$\endgroup$
  • $\begingroup$ Much of this looks at odds with the standard statistical literature, especially your definition of parameter. It appears to confound the processes of finding a procedure to calculate an estimate and identifying what is to be estimated. The latter--choosing the estimand--is a matter for the scientist or investigator to determine. The former is then selected by the statistician to have desirable properties among all possible procedures for estimating the estimand. There are also technical issues; suffice it to say that a parameter is more restricted than an arbitrary estimand. $\endgroup$ – whuber Jul 5 '13 at 13:44
  • $\begingroup$ I will expand my answer to address this. $\endgroup$ – Nick Cox Jul 5 '13 at 13:50
  • 1
    $\begingroup$ I agree with Tukey although you might think from my answer to this thread that I am one of the "ossified" statisticians he challenges. The problem is that you have taken his quotation out of context. Tukey is specifically addressing the question of how to evaluate the properties of procedures "when the hypotheses on which they are customarily developed do not hold." This in no way changes the definitions of things like parameters, estimators, and estimands. In particular, a parameter still is not "whatever we are estimating." $\endgroup$ – whuber Jul 5 '13 at 15:32
  • 3
    $\begingroup$ Much food for thought here. As a quick reply: My answer was not intended to imply that we are in Liberty Hall where anything goes. The context for the Tukey quotation I welcome, as my point of view is that it is usual that the customary hypotheses do not hold in so far as all models are approximations not exactly matched by the data. So far from biting, that clause underlines the value of the different point of view. In general, I am not attempting, nor qualified to produce, more abstract and more mathematically refined formal definitions. $\endgroup$ – Nick Cox Jul 5 '13 at 16:06
6
$\begingroup$

I tend to think of parameters by analogy by thinking about the normal distribution:
$$ \text{pdf}=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2}\frac{(x_i-\mu)^2}{\sigma^2}} $$ What's important to recognize about this function is that, as ugly as it is, I pretty much know what most of the parts are. For example, I know what the numbers $1$ and $2$ are, what $\pi$ is ($\approx 3.1415926$) and what $e$ is ($\approx 2.718281828$); I know what it means to square something or to take the square root of something--I basically know it all. Moreover, if I wanted to know the height of the function at some specific $X$ value, $x_i$, then I obviously know that value too. In other words, once I know that the above equation is what I need to be working with, I know everything there is to know, once I learn the values for $\boldsymbol\mu$ and $\boldsymbol\sigma^2$. Those values are the parameters. Specifically they are unknown constants that control the behavior of the distribution. Thus, for instance, if I wanted to know the $X$ value that corresponded to the $25^{\text{th}}\%$, I can determine that (or anything else about that distribution), after knowing $\mu$ and $\sigma^2$ (but not the other way around). The above equation privileges $\mu$ and $\sigma^2$ in a way that it does not for any other value.

Likewise, if I were working with an OLS multiple regression model, where the data generating process is assumed to be:
$$ Y=\beta_0 + \beta_1X_1 + \beta_2X_2 + \varepsilon \\ \text{where } \varepsilon\sim\mathcal N(0, \sigma^2) $$ then, once I learn (in practice, estimate) the values of $\boldsymbol\beta_0$, $\boldsymbol\beta_1$, $\boldsymbol\beta_2$, and $\boldsymbol\sigma^2$, I know everything there is to know. Anything else, such as the $25^{\text{th}}\%$ of the conditional distribution of $Y$ where $X=x_i$, I can calculate based on my knowledge of $\beta_0$, $\beta_1$, $\beta_2$, and $\sigma^2$. The multiple regression model above privileges $\beta_0$, $\beta_1$, $\beta_2$, and $\sigma^2$ in a way that it does not for any other value.

(All of this assumes, of course, that my model of the population distribution or data generating process is correct. It is, as always, worth bearing in mind that "all models are wrong, but some are useful" -George Box.)

To answer your questions more explicitly, I would say:

  • No, any old quantitative properly should not be labelled a "parameter".
  • n/a
  • The characteristics that should be labelled a "parameter" depend on the model specification. I don't have a special name for other quantitative characteristics, but I think it would be fine to call them properties or characteristics or consequences, etc.
$\endgroup$
  • $\begingroup$ Thanks. But what terminology do you use to describe all those population values that can be derived from a parametric model but are not in the set of convenient parameters for representing that model? Or alternatively, there might be a case, where you don't know the population model and don't particularly care, but are interested in a particular non-standard aspect of the population model. $\endgroup$ – Jeromy Anglim Jul 5 '13 at 2:45
  • $\begingroup$ I don't have any generally applicable special name, but there are names for some particular values. Eg, if you don't really believe that your population is sufficiently close to any well studied distribution, you could attempt to characterize it by its median, quartiles, hinge points, etc. $\endgroup$ – gung Jul 5 '13 at 2:53
  • 2
    $\begingroup$ A subtle issue about parameters is exposed by an old computer trick: take the binary (or decimal) representations of $\beta_0, \beta_1, \beta_2,$ and $\sigma$ and interleave them (by groups of four) to make a new binary (or decimal) number $\theta$. Obviously the process is reversible: you can read $\beta_0$ off the first, fifth, ninth, ..., etc. digits of $\theta$, and so on. Therefore "once [you] learn the value of $\theta$, [you] know everything there is to know." But $\theta$ is not a valid parameter due to the contorted way in which it labels possible distributions. $\endgroup$ – whuber Jul 5 '13 at 19:24
3
$\begingroup$

There have been some great answers to this question, I just thought I'd summarise an interesting reference that provides a fairly rigorous discussion of estimators.

The virtual laboratories page on estimators defines

  • a statistic as "an observable function of the outcome variable".
  • "in the technical sense, a parameter $\theta$ is a function of the distribution of X"

The concept of a function of a distribution is a very general idea. Thus, every example provided above could be seen as a function of a certain distribution.

  • Every quantile, including the min, median, 25th quantile, the max can be a function of a distribution.
  • Skewness is a function of a distribution. If that population distribution is normal, then these will be zero, but that does not stop the calculation of these values.
  • Counting the number of correlations greater than a certain value is a function of the covariance matrix which in turn is a function of a multivariate distribution.
  • R-squared is a function of the distribution.
$\endgroup$
  • 1
    $\begingroup$ One reason I offered a more elaborate answer is that this definition of "parameter" is not good enough. For one counterexample see my comment to @gung's answer. Intuitively, a set of parameterized distributions forms a finite-dimensional topological manifold-with-boundary; a parameter has to be a continuous function defined on the manifold. This is more than just a technical requirement, because it relates to the sampling distributions of estimates. $\endgroup$ – whuber Jul 7 '13 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.