Is any quantitative property of the population a "parameter"?

Question

I'm relatively familiar with the distinction between the terms statistic and parameter. I see a statistic as the value obtained from applying a function to the sample data. However, most examples of parameters relate to defining a parametric distribution. A common example is the mean and standard deviation to parameterise the normal distribution or the coefficients and error variance to parameterise a linear regression.

However, there are many other values of the population distribution that are less prototypical (e.g., minimum, maximum, r-square in multiple regression, the .25 quantile, median, the number of predictors with non-zero coefficients, skewness, the number of correlations in a correlation matrix greater than .3, etc.).

Thus, my questions are:

• Should any quantitative property of a population be labelled a "parameter"?
• If yes, then why?
• If no, what characteristics should not be labelled a parameter? What should they be labelled? And why?

Elaboration on confusion

The Wikipedia article on estimators states:

An "estimator" or "point estimate" is a statistic (that is, a function of the data) that is used to infer the value of an unknown parameter in a statistical model.

But I can define the unknown value as .25 quantile and I can develop an estimator for that unknown. I.e., not all quantitative properties of a population are parameters in the same way that say the mean and sd are parameters of a normal distribution, yet it is legitimate to seek to estimate any quantitative population property.

Answer 1

This question goes to the heart of what statistics is and how to to conduct a good statistical analysis. It raises many issues, some of terminology and others of theory. To clarify them, let's begin by noting the implicit context of the question and go on from there to define the key terms "parameter," "property," and "estimator." The several parts of the question are answered as they come up in the discussion. The final concluding section summarizes the key ideas.

State spaces

A common statistical use of "the distribution," as in "the Normal distribution with PDF proportional to $\exp(-\frac{1}{2}(x-\mu)/\sigma)^2)dx$" is actually a (serious) abuse of English, because obviously this is not one distribution: it's a whole family of distributions parameterized by the symbols $\mu$ and $\sigma$. A standard notation for this is the "state space" $\Omega$, a set of distributions. (I am simplifying a bit here for the sake of exposition and will continue to simplify as we go along, while remaining as rigorous as possible.) Its role is to delineate the possible targets of our statistical procedures: when we estimate something, we are picking out one (or sometimes more) elements of $\Omega$.

Sometimes state spaces are explicitly parameterized, as in $\Omega = \{\mathcal{N}(\mu, \sigma^2)|\mu \in \mathbb{R}, \sigma \gt 0\}$. In this description there is a one-to-one correspondence between the set of tuples $\{(\mu,\sigma)\}$ in the upper half plane and the set of distributions we will be using to model our data. One value of such a parameterization is that we may now refer concretely to distributions in $\Omega$ by means of an ordered pair of real numbers.

In other cases state spaces are not explicitly parameterized. An example would be the set of all unimodal continuous distributions. Below, we will address the question of whether an adequate parameterization can be found in such cases anyway.

Parameterizations

Generally, a parameterization of $\Omega$ is a correspondence (mathematical function) from a subset of $\mathbb{R}^d$ (with $d$ finite) to $\Omega$. That is, it uses ordered sets of $d$-tuples to label the distributions. But it's not just any correspondence: it has to be "well behaved." To understand this, consider the set of all continuous distributions whose PDFs have finite expectations. This would widely be regarded as "non-parametric" in the sense that any "natural" attempt to parameterize this set would involve a countable sequence of real numbers (using an expansion in any orthogonal basis). Nevertheless, because this set has cardinality $\aleph_1$, which is the cardinality of the reals, there must exist some one-to-one correspondence between these distributions and $\mathbb{R}$. Paradoxically, that would seem to make this a parameterized state space with a single real parameter!

The paradox is resolved by noting that a single real number cannot enjoy a "nice" relationship with the distributions: when we change the value of that number, the distribution it corresponds to must in some cases change in radical ways. We rule out such "pathological" parameterizations by requiring that distributions corresponding to close values of their parameters must themselves be "close" to one another. Discussing suitable definitions of "close" would take us too far afield, but I hope this description is enough to demonstrate that there is much more to being a parameter than just naming a particular distribution.

Properties of distributions

Through repeated application, we become accustomed to thinking of a "property" of a distribution as some intelligible quantity that frequently appears in our work, such as its expectation, variance, and so on. The problem with this as a possible definition of "property" is that it's too vague and not sufficiently general. (This is where mathematics was in the mid-18th century, where "functions" were thought of as finite processes applied to objects.) Instead, about the only sensible definition of "property" that will always work is to think of a property as being a number that is uniquely assigned to every distribution in $\Omega$. This includes the mean, the variance, any moment, any algebraic combination of moments, any quantile, and plenty more, including things that cannot even be computed. However, it does not include things that would make no sense for some of the elements of $\Omega$. For instance, if $\Omega$ consists of all Student t distributions, then the mean is not a valid property for $\Omega$ (because $t_1$ has no mean). This impresses on us once again how much our ideas depend on what $\Omega$ really consists of.

Properties are not always parameters

A property can be such a complicated function that it would not serve as a parameter. Consider the case of the "Normal distribution." We might want to know whether the true distribution's mean, when rounded to the nearest integer, is even. That's a property. But it will not serve as a parameter.

Parameters are not necessarily properties

When parameters and distributions are in one-to-one correspondence then obviously any parameter, and any function of the parameters for that matter, is a property according to our definition. But there need not be a one-to-one correspondence between parameters and distributions: sometimes a few distributions must be described by two or more distinctly different values of the parameters. For instance, a location parameter for points on the sphere would naturally use latitude and longitude. That's fine--except at the two poles, which correspond to a given latitude and any valid longitude. The location (point on the sphere) indeed is a property but its longitude is not necessarily a property. Although there are various dodges (just declare the longitude of a pole to be zero, for instance), this issue highlights the important conceptual difference between a property (which is uniquely associated with a distribution) and a parameter (which is a way of labeling the distribution and might not be unique).

Statistical procedures

The target of an estimate is called an estimand. It is merely a property. The statistician is not free to select the estimand: that is the province of her client. When someone comes to you with a sample of a population and asks for you to estimate the population's 99th percentile, you would likely be remiss in supplying an estimator of the mean instead! Your job, as statistician, is to identify a good procedure for estimating the estimand you have been given. (Sometimes your job is to persuade your client that he has selected the wrong estimand for his scientific objectives, but that's a different issue...)

By definition, a procedure is a way to get a number out of the data. Procedures are usually given as formulas to be applied to the data, like "add them all up and divide by their count." Literally any procedure may be pronounced an "estimator" of a given estimand. For instance, I could declare that the sample mean (a formula applied to the data) estimates the population variance (a property of the population, assuming our client has restricted the set of possible populations $\Omega$ to include only those that actually have variances).

Estimators

An estimator needn't have any obvious connection to the estimand. For instance, do you see any connection between the sample mean and a population variance? Neither do I. But nevertheless, the sample mean actually is a decent estimator of the population variance for certain $\Omega$ (such as the set of all Poisson distributions). Herein lies one key to understanding estimators: their qualities depend on the set of possible states $\Omega$. But that's only part of it.

A competent statistician will want to know how well the procedure they are recommending will actually perform. Let's call the procedure "$t$" and let the estimand be $\theta$. Not knowing which distribution actually is the true one, she will contemplate the procedure's performance for every possible distribution $F \in \Omega$. Given such an $F$, and given any possible outcome $s$ (that is, a set of data), she will compare $t(s)$ (what her procedure estimates) to $\theta(F)$ (the value of the estimand for $F$). It is her client's responsibility to tell her how close or far apart those two are. (This is often done with a "loss" function.) She can then contemplate the expectation of the distance between $t(s)$ and $\theta(F)$. This is the risk of her procedure. Because it depends on $F$, the risk is a function defined on $\Omega$.

(Good) statisticians recommend procedures based on comparing risk. For instance, suppose that for every $F \in \Omega$, the risk of procedure $t_1$ is less than or equal to the risk of $t$. Then there is no reason ever to use $t$: it is "inadmissible." Otherwise it is "admissible".

(A "Bayesian" statistician will always compare risks by averaging over a "prior" distribution of possible states (usually supplied by the client). A "Frequentist" statistician might do this, if such a prior justifiably exists, but is also willing to compare risks in other ways Bayesians eschew.)

Conclusions

We have a right to say that any $t$ that is admissible for $\theta$ is an estimator of $\theta$. We must, for practical purposes (because admissible procedures can be hard to find), bend this to saying that any $t$ that has acceptably small risk (when being compared to $\theta$) among practicable procedures is an estimator of $\theta$. "Acceptably" and "practicable" are determined by the client, of course: "acceptably" refers to their risk and "practicable" reflects the cost (ultimately paid by them) of implementing the procedure.

Underlying this concise definition are all the ideas just discussed: to understand it we must have in mind a specific $\Omega$ (which is a model of the problem, process, or population under study), a definite estimand (supplied by the client), a specific loss function (which quantitatively connects $t$ to the estimand and is also given by the client), the idea of risk (computed by the statistician), some procedure for comparing risk functions (the responsibility of the statistician in consultation with the client), and a sense of what procedures actually can be carried out (the "practicability" issue), even though none of these are explicitly mentioned in the definition.

Answer 2

As with many questions on definitions, answers need to have an eye both on the underlying principles and on the ways terms are used in practice, which can often be at least a little loose or inconsistent, even by individuals who are well informed, and more importantly, variable from community to community.

One common principle is that a statistic is a property of a sample, and a known constant, and a parameter is the corresponding property of the population, and so an unknown constant. The word "corresponding" is to be understood as quite elastic here. Incidentally, precisely this distinction and precisely this terminology are less than a century old, having being introduced by R.A. Fisher.

But

1. A set-up of sample and population doesn't characterise all our own problems. Time series are one major class of examples in which the idea is rather of an underlying generating process, and something like that is arguably the deeper and more general idea.

2. There are set-ups in which parameters change. Again, time series analysis provides examples.

3. To the main point here, we don't in practice think of all the properties of a population or process as parameters. If some procedure assumes a model of a normal distribution, then the minimum and maximum are not parameters. (Indeed, according to the model, the minimum and maximum are arbitrarily large negative and positive numbers any way, not that that should worry us.)

I would say that for once Wikipedia is pointing in the right direction here, and practice and principle are both respected if we say that a parameter is whatever we are estimating.

This helps too with other questions that have caused puzzlement. For example, if we calculate a 25% trimmed mean, what we are estimating? A reasonable answer is the corresponding property of the population, which in effect is defined by the estimation method. One terminology is that an estimator has an estimand, whatever it is estimating. Starting with some Platonic idea of a property "out there" (say the mode of a distribution) and thinking how to estimate that is reasonable, as is thinking up good recipes for analysing data and thinking through what they imply when regarded as inference.

As often in applied mathematics or science, there is a twofold aspect to a parameter. We often think of it as something real out there which we are discovering, but it is also true that it is something defined by our model of the process, so that it has no meaning outside the context of the model.

Two quite different points:

1. Many scientists use the word "parameter" in the way that statisticians use variable. I have a scientist persona as well as a statistical one, and I would say that is unfortunate. Variables and properties are better words.

2. It is remarkably common in wider English usage that parameter is thought to mean limits or bounds, which may stem from some original confusion between "parameter" and "perimeter".

A note on the estimand point of view

The classical position is that we identify a parameter in advance and then decide how to estimate it, and this remains majority practice, but reversing the process is not absurd and can be helpful for some problems. I call this the estimand point of view. It has been in the literature for at least 50 years. Tukey (1962, p.60) urged that

"We must give even more attention to starting with an estimator and discovering what is a reasonable estimand, to discovering what is it reasonable to think of the estimator as estimating."

A similar point of view has been elaborated formally in considerable detail and depth by Bickel and Lehmann (1975) and informally with considerable lucidity by Mosteller and Tukey (1977, pp.32-34).

There is also an elementary version. Using (say) sample median or geometric mean to estimate the corresponding population parameter makes sense regardless of whether the underlying distribution is symmetric, and the same goodwill can be extended to (e.g.) sample trimmed means, which are regarded as estimators of their population counterparts.

Bickel, P.J. and E.L. Lehmann. 1975. Descriptive statistics for nonparametric models. II. Location. Annals of Statistics 3: 1045-1069.

Mosteller, F. and J.W. Tukey. 1977. Data Analysis and Regression. Reading, MA: Addison-Wesley.

Tukey, J.W. 1962. The future of data analysis. Annals of Mathematical Statistics 33: 1-67.

Answer 3

I tend to think of parameters by analogy by thinking about the normal distribution:
$$ \text{pdf}=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{1}{2}\frac{(x_i-\mu)^2}{\sigma^2}} $$ What's important to recognize about this function is that, as ugly as it is, I pretty much know what most of the parts are. For example, I know what the numbers $1$ and $2$ are, what $\pi$ is ($\approx 3.1415926$) and what $e$ is ($\approx 2.718281828$); I know what it means to square something or to take the square root of something--I basically know it all. Moreover, if I wanted to know the height of the function at some specific $X$ value, $x_i$, then I obviously know that value too. In other words, once I know that the above equation is what I need to be working with, I know everything there is to know, once I learn the values for $\boldsymbol\mu$ and $\boldsymbol\sigma^2$. Those values are the parameters. Specifically they are unknown constants that control the behavior of the distribution. Thus, for instance, if I wanted to know the $X$ value that corresponded to the $25^{\text{th}}\%$, I can determine that (or anything else about that distribution), after knowing $\mu$ and $\sigma^2$ (but not the other way around). The above equation privileges $\mu$ and $\sigma^2$ in a way that it does not for any other value.

Likewise, if I were working with an OLS multiple regression model, where the data generating process is assumed to be:
$$ Y=\beta_0 + \beta_1X_1 + \beta_2X_2 + \varepsilon \\ \text{where } \varepsilon\sim\mathcal N(0, \sigma^2) $$ then, once I learn (in practice, estimate) the values of $\boldsymbol\beta_0$, $\boldsymbol\beta_1$, $\boldsymbol\beta_2$, and $\boldsymbol\sigma^2$, I know everything there is to know. Anything else, such as the $25^{\text{th}}\%$ of the conditional distribution of $Y$ where $X=x_i$, I can calculate based on my knowledge of $\beta_0$, $\beta_1$, $\beta_2$, and $\sigma^2$. The multiple regression model above privileges $\beta_0$, $\beta_1$, $\beta_2$, and $\sigma^2$ in a way that it does not for any other value.

(All of this assumes, of course, that my model of the population distribution or data generating process is correct. It is, as always, worth bearing in mind that "all models are wrong, but some are useful" -George Box.)

To answer your questions more explicitly, I would say:

• No, any old quantitative properly should not be labelled a "parameter".
• n/a
• The characteristics that should be labelled a "parameter" depend on the model specification. I don't have a special name for other quantitative characteristics, but I think it would be fine to call them properties or characteristics or consequences, etc.

Answer 4

There have been some great answers to this question, I just thought I'd summarise an interesting reference that provides a fairly rigorous discussion of estimators.

The virtual laboratories page on estimators defines

• a statistic as "an observable function of the outcome variable".
• "in the technical sense, a parameter $\theta$ is a function of the distribution of X"

The concept of a function of a distribution is a very general idea. Thus, every example provided above could be seen as a function of a certain distribution.

• Every quantile, including the min, median, 25th quantile, the max can be a function of a distribution.
• Skewness is a function of a distribution. If that population distribution is normal, then these will be zero, but that does not stop the calculation of these values.
• Counting the number of correlations greater than a certain value is a function of the covariance matrix which in turn is a function of a multivariate distribution.
• R-squared is a function of the distribution.