Understanding expectation of data points estimators

Question

I have a small doubt in the way estimators are defined. We have a sequence of data points $X^{(1)}, X^{(2)}, ... , X^{(n)}$. We then define an estimator as a function of these data points. When calculating the bias of an estimator we find the expectation of this function of "random variables".

But are $X^{(1)}, X^{(2)} ... $ all random variables? These are just observed values. I know the definition says they are samples coming from some unknown distribution (at least the parameter of the distribution is not known). How can we possibly compute $E[X^{(i)}]$ when we do not know what the distribution is? Also I tend to see each $X^{(i)}$ as an observed data value of the random variable and not a random variable. What is it that I am missing?

Many proofs that I have seen (like the one that proves $\bar{X}$ is unbiased) end up using the iid assumption to get rid of the $E[X^{(i)}]$. None of them talks about calculating expectation explicitly. Any more examples to show what is meant by calculating the expectation of values of random variables and saying that these are iid could be helpful in understanding the concepts.

Update:

One way I am beginning to think of it is the following:

We have a set of n data points corresponding to a random variable. Now instead of looking at these as n distinct samples of one random variable, define n iid random variables having the same distribution as the original random variables and consider these to be the n values of these n random variables. Am I thinking in the right direction?

Answer 1

What is a random variable? It is a real valued measurable function: $X:(\Omega,\mathcal{F})\to (\mathbb{R},\mathcal{B}(\mathbb{R}))$. What this means is that for each $\omega \in \Omega$, $X$ gives us real number $x$.

When we have a sample we observe one realisation, i.e. we have values of $X^{(i)}$ for one $\omega$: $X^{(i)}(\omega)=x^{i}$. If we were able to repeat the process for all $\omega$ we would get all the possible values of $X^{(i)}$. So instead of one realisation we would have a lot of realisations $\{X^{(i)}(\omega),\omega\in\Omega\}$. So when we treat data point as a random variable we actualy have in mind all of these realisations. Treating them as random variables is convenient because when we say something about random variable we actualy say something about all the realisations.

Now when we treat $X^{(i)}$ as random variable we can calculate its expectation only if we have its distribution. But this is where models come in. Usually we assume something about our random variables, or that is equivalent to assuming something about the data. One of the possible assumptions is that expectation of random variable exists. For example let us say that our sample is iid sample. This means that each $X^{(i)}$ is a random variable with distribution function $F$ and these random variables are independent of each other. Then we can explicitly calculate the expectation of $X^{(i)}$:

$$EX^{(i)}=\int_{-\infty}^{\infty}xdF(x)$$

If we assume that $F$ is of normal distribution with parameters $\mu$ and $\sigma^2$, then we can plug in the $F$ into this formula and get that $EX^{(i)}=\mu$.

Answer 2

In probability theory, the expected value (or expectation, or mathematical expectation, or mean, or the first moment) of a random variable is the weighted average of all possible values that this random variable can take on.

Example of Die

Let X represent the outcome of a roll of a six-sided die. More specifically, X will be the number of pips showing on the top face of the die after the toss. The possible values for X are 1, 2, 3, 4, 5, 6, all equally likely (each having the probability of 1/6). The expectation of X is

$$E(X)=1\cdot(1/6)+2\cdot(1/6)+3\cdot(1/6)+4\cdot(1/6)+5\cdot(1/6)+6\cdot(1/6)=21/6$$

If you roll the die n times and compute the average (mean) of the results, then as n grows, the average will almost surely converge to the expected value, a fact known as the strong law of large numbers. One example sequence of ten rolls of the die is 2, 3, 1, 2, 5, 6, 2, 2, 2, 6, which has the average of 3.1, with the distance of 0.4 from the expected value of 3.5. The convergence is relatively slow: the probability that the average falls within the range 3.5 ± 0.1 is 21.6% for ten rolls, 46.1% for a hundred rolls and 93.7% for a thousand rolls. See the figure for an illustration of the averages of longer sequences of rolls of the die and how they converge to the expected value of 3.5.

Moreover, the weights used in computing this average correspond to the probabilities in case of a discrete random variable, or densities in case of a continuous random variable.