Distribution of the point estimator of a sample

Question

We know that point estimator is defined over a sample and is said to be unbiased if its expectation value is same as that of some parametric function $g(\theta)$ where $\theta$ is a parameter for the probability density function for the population which is unknown.

Suppose we have a sample of size $n$, i.e., $\underline X=(X_1,\;X_2,\;\ldots,\;X_n)$ i.e., $n$ iid random variables.
Then the expectation of sample mean,
$E(\bar X)=\frac{\sum_{i=1}^nE(X_i)}{n}=\mu=$ population mean
Similarly, if we take an extimator as $T(\underline X)=X_1$, then also $E(T(\underline X))=E(X_1)=\mu$
So, we can see that the point estimator itself is a random variable, and it has a distribution.
In the above proofs, we are taking expectation of the estimator. Expectation is meaningful if we know the distribution.
But for each random variable $X_i's$, their distribution is same as that of the population which is known and which we are trying to estimate.

Doubt

So suppose we have infinitely many samples i.e. all the possible values for $\underline X$ or $range(\underline X)$, this means we have all possible values for the estimator also i.e., $range(\bar X)$.
Given this, I am not able to figure out, how we calculate the expectation of $\bar X$? (We don't know the distribution of $\bar X$ as distribution of each $X_i$ is unknown). The above proofs for unbiased estimators suggests that if we calculate that expectation, this will coincide with the population mean.

Addendum
To make my doubt more precise, I am adding an example.
Consider a binomial distribution of 5 trials with the probability of success $(\theta)$ being unknown. So the sample space can be considered in the form of random variable $X$ with $range(X)={\{0,1,2,3,4,5}\}$.
Now the aim of the estimator is to estimate some $g(\theta)$. Now, we can take all the possible sample of size 3 and calculate two different estimators as $T_1((X_1,X_2,X_3)=\frac{X_1+X_2+X_3}{3}$ and $T_2((X_1,X_2,X_3))=X_1$

i) Now, I have a doubt that how can we calculate $E(T_1)$ and $E(T_2)$? Should we have to consider uniform distribution? But why, if yes?

ii) Also, if we consider some distribution on the estimator and calculate its expectation value, it will be independent of $\theta$, then how can we estimate $g(\theta)$?

Answer 1

There is an impending confusion lurking throughout the post. Let me leave a brief generic discussion.

Consider the sample sapce $(\Omega, \boldsymbol{\mathfrak A}, \mathbb P).$ Let $\mathbf X:\Omega\to\mathbb R^n$ be a random $n$-vector. Let the induced probability measure or distribution on $\boldsymbol{\mathfrak B}(\mathbb R^n)$ be denoted by $\mathbb P_\mathbf X. $ Now, let $\boldsymbol f:\mathbb R^n\to \mathbb R^p, ~\varphi:\mathbb R^p \to \mathbb R$ be two Borel functions.

Then, if $\varphi$ is $\mathbb P_{\boldsymbol f(\mathbf X) }$ measurable, as shown here,

$$\int_{\mathbb R^p}\varphi(\boldsymbol t) ~\mathrm d\mathbb P_{\boldsymbol f(\mathbf X) }(\boldsymbol t) =\int_{\mathbb R^n} (\varphi\circ\boldsymbol f) (\boldsymbol s) ~\mathrm d \mathbb P_{\mathbf X}(\boldsymbol s)\tag 1\label 1.$$

Taking $f:\mathbb R^n\to \mathbb R;~\varphi: \mathbb R\to\mathbb R$ as $ s\mapsto s, $ from $\eqref 1,$

$$ \mathbb E[f(\mathbf X) ]=\int f(\boldsymbol s) ~\mathrm d\mathbb P_\mathbf X(\boldsymbol s) .\tag 2\label 2$$

---

A sample of size $n$ is a statistically independent sequence of random variables, that is, a sample consists of random vector $\mathbf X\in \mathbb R^n.$ Then $\mathbb P_\mathbf X= \mathbb P_{X_1}\otimes \cdots \otimes \mathbb P_{X_n}.$ A statistic is a Borel function $g:\mathbb R^n\to \mathbb R .$ Let $g$ be $\mathbf X\mapsto\frac{ \sum_{1\leq i\leq n} X_i}{n}.$ By $\eqref 2,$ \begin{align} \mathbb E[g(\mathbf X) ]&=\int g ~\mathrm d\mathbb P_\mathbf X\\&=\frac1n \int\left(\sum_{1\leq i\leq n} X_i\right)~\mathrm d(\mathbb P_{X_1}\otimes \cdots \otimes \mathbb P_{X_n})\\&\overset{\eqref 1}{=}\frac1n\sum_{1\leq i\leq n}\int X_i~\mathrm d\mathbb P_{X_i}\\&=\frac1n\sum_{1\leq i\leq n}\mathbb E[X_i].\tag 3\label 3\end{align}

Since in a sample, all the rvs are identically distributed, $\eqref 3$ yields the familiar result of unbiasedness of sample mean.

---

how we calculate the expectation of $\bar X$? (We don't know the distribution of $\bar X$ as distribution of each $X_i$ is unknown)

I couldn't understand the question. Consider the parametric model $X_i\sim P_\theta.$ We are aware of the structure of the distribution but we don't know about the index parameter. So, can't we calculate the expectation if a statistic is based on sample from $P_\theta? $ Perhaps you are obfuscating what can be calculated from what is already known and what cannot be computed since the specific $\theta$ is not known.

So, let's talk about the $T_1$ and $T_2.$ Assume $X_i\sim\mathrm{Bin}(5, \theta). $

What should be $\mathbb E[T_1]$? If one goes along the line of $\eqref 3,$ one can succinctly write $$ \mathbb E[T_1]=\frac1{3}(\mathbb E[X_1]+\mathbb E[X_2]+\mathbb E[X_3])=5\theta.$$ Similarly, one can calculate $\mathbb E[T_2].$ We don't know the exact distribution and yet with formal application of change of variables, linearity, we could calculate the integral or expectation.

Also, if we consider some distribution on the estimator and calculate its expectation value, it will be independent of $θ$, then how can we estimate $g(θ)$?

Again I am unable to make sense of this particular choice of words consider some distribution on the estimator. If you choose a certain function $g(\mathbf X), $ it is already ascertained what distribution it would follow irrespective of whether you can evaluate feasibly or not.

Recall the standard result, for instance, that $\bar X\sim \mathcal N(u, \sigma^2/n) $ when $X_i\sim\mathcal N(\mu, \sigma^2). $ When you chose the estimator to be the sample mean, it was already ascertained, due to what the parent population was, what $\bar X$ would follow.

and calculate its expectation value, it will be independent of θ, then how can we estimate g(θ) - the statistic is by definition not dependent on the parameter. But I could not comprehend what you meant by the expectation value being independent of $\theta.$

---

Reference:

$\rm [I]$ Mathematical Statistics, Wiebe R. Pestman, Walter de Gruyter, $2009, $ sec. $1.4-1.5.$