What is the 'true' value of a probability parameter?

Question

We seem to distinguish empirical estimates of parameters from 'true' values, and make comparisons between the two. I can understand what an empirical estimate is. What is a 'true' value?

For instance, my course notes have:

Definition 4.7 (Consistent estimator) An estimator $T(X_1,...,X_n)$ for a parameter $\theta$ is consistent if, for any $\epsilon>0$, we have $$ \lim_{n\rightarrow\infty}P(|T(X_1,...,X_n)-\theta|<\epsilon)=1 $$

How do we determine $\theta$ apart from by estimating it, in order to compare it to an estimator?

Answer 1

In general the "true value" is a fiction, defined within a model that in reality won't fit perfectly, in which case consequently there is also no such thing as a "true value". Assuming that there is a true parameter value is a device for doing theory and developing methods. It allows us to theoretically show that this-or-that method to estimate it has this-or-that property and works better or worse, which is a motivation for these methods even though it doesn't correspond exactly to the real situation (but then no model does).

If we simulate artificial data, however, we can fix and control the true parameter values, in which case we can compare the estimate to the true value (ignoring here potential issues with random number generation).

There are also some real situations in which true values can be "controlled" or known to some extent, for example if we have a measurement instrument that is meant to measure a certain quantity with stochastic measurement error, and in some situations we may be able to control the quantity that is measured. This still cannot guarantee the truth of the measurement error model within which the model parameter is defined, but at least we can control the real quantity that is interpreted to correspond to the true parameter value. An example for this are indirect estimates from age determination methods applied to individuals of which we know the precise age.

Sometimes we estimate parameters that correspond to existing population quantities of a usually big but finite population (such as population means of something that is well defined for all population members such as age) from a sample, in which case the population quantity would correspond to the "true" parameter value.

Answer 2

It all boils down to theory vs. practice.

The true value of a parameter is always a theoretical quantity. Thus, you can never determine the true $\theta$.

The idea behind this is, that there is some kind of process, which generates the data. This process has some parameter. If you knew that parameter and the process you could generate data, which would be indistinguishable from the real process. Other parameters always generate data, which can eventually be distinguished from the real process.

The only way to "know" the true $\theta$ is if you define the process mathematically. For example, if you define a fair coin, then the true probability of showing heads is $0.5$, but not because you determine it by an experiment, but because you defined it to be.

A consistent estimator will tend (probabilistically!) towards the true value, which is captured by the definition you provided. As you collect more data, the probability that the consistent estimator differs by a certain amount $\epsilon$ from the true value goes towards zero (NOTE: it is a common misconception that probability of zero means something never happens, but that is not necessarily true. It only means, it practically never happens).

So if you want to "determine" a true value just based on the data alone, a consistent estimator is your best method. You can never be sure, that you really got the right value (which is why any estimator should come with confidence intervals, etc).

Now, how do you know an estimator is a consistent estimator if you can't know the true value in practice? If some estimator is consistent is not a practical observation, but a theoretical property. Thus, you prove (theory!) that something is a consistent estimator.

Take for example the expected value and variance (true parameters) of a normal distributed variable (the process). The mean of the samples is a consistent estimator of the expected value. So you define $X \sim Normal(\mu,\sigma^2)$ (i.e. $\theta = (\mu,\sigma)$), and then you prove (!) that the mean will tend probabilistically towards $\mu$ for any $\mu$ and $\sigma$. Then you know, that the method you have just proven can be applied in practice to estimate the true $\mu$.