# Distribution of the point estimator of a sample

We know that point estimator is defined over a sample and is said to be unbiased if its expectation value is same as that of some parametric function $$g(\theta)$$ where $$\theta$$ is a parameter for the probability density function for the population which is unknown.

Suppose we have a sample of size $$n$$, i.e., $$\underline X=(X_1,\;X_2,\;\ldots,\;X_n)$$ i.e., $$n$$ iid random variables.
Then the expectation of sample mean,
$$E(\bar X)=\frac{\sum_{i=1}^nE(X_i)}{n}=\mu=$$ population mean
Similarly, if we take an extimator as $$T(\underline X)=X_1$$, then also $$E(T(\underline X))=E(X_1)=\mu$$
So, we can see that the point estimator itself is a random variable, and it has a distribution.
In the above proofs, we are taking expectation of the estimator. Expectation is meaningful if we know the distribution.
But for each random variable $$X_i's$$, their distribution is same as that of the population which is known and which we are trying to estimate.

Doubt

So suppose we have infinitely many samples i.e. all the possible values for $$\underline X$$ or $$range(\underline X)$$, this means we have all possible values for the estimator also i.e., $$range(\bar X)$$.
Given this, I am not able to figure out, how we calculate the expectation of $$\bar X$$? (We don't know the distribution of $$\bar X$$ as distribution of each $$X_i$$ is unknown). The above proofs for unbiased estimators suggests that if we calculate that expectation, this will coincide with the population mean.

To make my doubt more precise, I am adding an example.
Consider a binomial distribution of 5 trials with the probability of success $$(\theta)$$ being unknown. So the sample space can be considered in the form of random variable $$X$$ with $$range(X)={\{0,1,2,3,4,5}\}$$.
Now the aim of the estimator is to estimate some $$g(\theta)$$. Now, we can take all the possible sample of size 3 and calculate two different estimators as $$T_1((X_1,X_2,X_3)=\frac{X_1+X_2+X_3}{3}$$ and $$T_2((X_1,X_2,X_3))=X_1$$

i) Now, I have a doubt that how can we calculate $$E(T_1)$$ and $$E(T_2)$$? Should we have to consider uniform distribution? But why, if yes?

ii) Also, if we consider some distribution on the estimator and calculate its expectation value, it will be independent of $$\theta$$, then how can we estimate $$g(\theta)$$?

• Re "Expectation is meaningful if we know the distribution." This appears to confuse an epistemological or psychological idea with a mathematical one. In mathematics the expectation is meaningful whether or not you happen to know the distribution. You can calculate the expectations by using mathematical facts about them: linearity is the most fundamental one.
– whuber
Commented Feb 6, 2023 at 15:09

There is an impending confusion lurking throughout the post. Let me leave a brief generic discussion.

Consider the sample sapce $$(\Omega, \boldsymbol{\mathfrak A}, \mathbb P).$$ Let $$\mathbf X:\Omega\to\mathbb R^n$$ be a random $$n$$-vector. Let the induced probability measure or distribution on $$\boldsymbol{\mathfrak B}(\mathbb R^n)$$ be denoted by $$\mathbb P_\mathbf X.$$ Now, let $$\boldsymbol f:\mathbb R^n\to \mathbb R^p, ~\varphi:\mathbb R^p \to \mathbb R$$ be two Borel functions.

Then, if $$\varphi$$ is $$\mathbb P_{\boldsymbol f(\mathbf X) }$$ measurable, as shown here,

$$\int_{\mathbb R^p}\varphi(\boldsymbol t) ~\mathrm d\mathbb P_{\boldsymbol f(\mathbf X) }(\boldsymbol t) =\int_{\mathbb R^n} (\varphi\circ\boldsymbol f) (\boldsymbol s) ~\mathrm d \mathbb P_{\mathbf X}(\boldsymbol s)\tag 1\label 1.$$

Taking $$f:\mathbb R^n\to \mathbb R;~\varphi: \mathbb R\to\mathbb R$$ as $$s\mapsto s,$$ from $$\eqref 1,$$

$$\mathbb E[f(\mathbf X) ]=\int f(\boldsymbol s) ~\mathrm d\mathbb P_\mathbf X(\boldsymbol s) .\tag 2\label 2$$

A sample of size $$n$$ is a statistically independent sequence of random variables, that is, a sample consists of random vector $$\mathbf X\in \mathbb R^n.$$ Then $$\mathbb P_\mathbf X= \mathbb P_{X_1}\otimes \cdots \otimes \mathbb P_{X_n}.$$ A statistic is a Borel function $$g:\mathbb R^n\to \mathbb R .$$ Let $$g$$ be $$\mathbf X\mapsto\frac{ \sum_{1\leq i\leq n} X_i}{n}.$$ By $$\eqref 2,$$ \begin{align} \mathbb E[g(\mathbf X) ]&=\int g ~\mathrm d\mathbb P_\mathbf X\\&=\frac1n \int\left(\sum_{1\leq i\leq n} X_i\right)~\mathrm d(\mathbb P_{X_1}\otimes \cdots \otimes \mathbb P_{X_n})\\&\overset{\eqref 1}{=}\frac1n\sum_{1\leq i\leq n}\int X_i~\mathrm d\mathbb P_{X_i}\\&=\frac1n\sum_{1\leq i\leq n}\mathbb E[X_i].\tag 3\label 3\end{align}

Since in a sample, all the rvs are identically distributed, $$\eqref 3$$ yields the familiar result of unbiasedness of sample mean.

how we calculate the expectation of $$\bar X$$? (We don't know the distribution of $$\bar X$$ as distribution of each $$X_i$$ is unknown)

I couldn't understand the question. Consider the parametric model $$X_i\sim P_\theta.$$ We are aware of the structure of the distribution but we don't know about the index parameter. So, can't we calculate the expectation if a statistic is based on sample from $$P_\theta?$$ Perhaps you are obfuscating what can be calculated from what is already known and what cannot be computed since the specific $$\theta$$ is not known.

So, let's talk about the $$T_1$$ and $$T_2.$$ Assume $$X_i\sim\mathrm{Bin}(5, \theta).$$

What should be $$\mathbb E[T_1]$$? If one goes along the line of $$\eqref 3,$$ one can succinctly write $$\mathbb E[T_1]=\frac1{3}(\mathbb E[X_1]+\mathbb E[X_2]+\mathbb E[X_3])=5\theta.$$ Similarly, one can calculate $$\mathbb E[T_2].$$ We don't know the exact distribution and yet with formal application of change of variables, linearity, we could calculate the integral or expectation.

Also, if we consider some distribution on the estimator and calculate its expectation value, it will be independent of $$θ$$, then how can we estimate $$g(θ)$$?

Again I am unable to make sense of this particular choice of words consider some distribution on the estimator. If you choose a certain function $$g(\mathbf X),$$ it is already ascertained what distribution it would follow irrespective of whether you can evaluate feasibly or not.

Recall the standard result, for instance, that $$\bar X\sim \mathcal N(u, \sigma^2/n)$$ when $$X_i\sim\mathcal N(\mu, \sigma^2).$$ When you chose the estimator to be the sample mean, it was already ascertained, due to what the parent population was, what $$\bar X$$ would follow.

and calculate its expectation value, it will be independent of θ, then how can we estimate g(θ) - the statistic is by definition not dependent on the parameter. But I could not comprehend what you meant by the expectation value being independent of $$\theta.$$

## Reference:

$$\rm [I]$$ Mathematical Statistics, Wiebe R. Pestman, Walter de Gruyter, $$2009,$$ sec. $$1.4-1.5.$$

• Thanks a lot for the answer. I have one doubt that suppose if we don't know about the distribution of population itself (Like in the given answer if we don't know that $X\sim Bin(N,\theta)$), then does in that case can we calculate expectation of point estimator and make some claim about the distribution of the population.
– Iti
Commented Feb 7, 2023 at 3:14
• Indeed you can work with expectation to reach viable results even if you are devoid of the knowledge of the specific family of distributions the sample has come from. See the derivation of unbiasedness of sample mean for population mean. In nowhere did we impose any additional specification of the distribution. Commented Feb 7, 2023 at 3:27
• I have one last doubt. Suppose $X\sim Bin(5,p)$. And we are taking sample $(\underline X=(X_1,X_2,X_3))$ of size 3. So, the $card(range(\underline X))=5^3=125$. Suppose I have all these 125 samples. Now, my question is for each sample we have sample mean and also suppose that we don't know that X has binomial distribution. In that case, how can we calculate the expectation of these sample means?
– Iti
Commented Feb 7, 2023 at 3:33
• After all these discussions, I feel that if we don't know the distribution of $X$ (population), then we should have infinitely many samples because then the frequency of each particular tuple in a sample tends to the actual probability of the that tuple (due to the underlying distribution of X) and then my above question of expectation of sample means is meaningful? Am I correct?
– Iti
Commented Feb 7, 2023 at 3:36
• Re sample mean, you really don't need to know the distribution, as I reiterated. But suppose you have an arbitrary Borel function $h:\mathbb R^n\to \mathbb R.$ How would you calculate $\mathbb E[h(\mathbf X)] =\int h ~\mathrm d\mathbb P_\mathbf X$? Unfortunately you can't do much here. Expectation is always meaningful but how much you can proceed depends on $h.$ Commented Feb 7, 2023 at 3:50