How to calculate the expected value of an estimator?

According to my book :

An estimator, say, T, of the parameter $$\theta$$ is said to be an unbiased estimator of $$\theta$$ if $$E\left( T\right) = \theta$$.

It then explains how to calculate $$E\left( T\right)$$ as follows:

$$E\left( T\right)$$ is obtained by taking the average value of $$T$$ computed from all possible samples of a given size that may be drawn from the population. That is, $$E\left( T\right) =\mu _{T}$$.

I am looking for an explanation of this method of calculating $$E\left( T\right)$$.

• You calculate the integral or the sum, same as calculating the expectation of any other random variable.
– Dave
Oct 22, 2020 at 21:53
• For general audience: Here and below my answer there were many comments trying to clear up OP's confusion over a passage in a biostat book. Each of us has deleted several now-stale comments. // What is now the first part of my answer is the latest part posted. Oct 23, 2020 at 5:59

Based on our discussion in comments, here is an attempt to clarify the discussion about estimators in your biostatistics text.

Suppose you have $$n = 25$$ observations from the normal distribution $$\mathsf{Norm}(\mu = 100, \sigma=15).$$ These might be test scores. Three possible estimators of the population mean are the sample mean (or average) $$A =\bar X = \frac 1n\sum X_i,$$ the sample median $$H$$ and the sample midrange $$D,$$ which is the average of the sample maximum and minimum.

All three of these estimators are unbiased, that is $$E(A) = E(H)= E(D) = \mu.$$ If we take one sample of size $$n = 25,$$ let's see what numerical results we get using these estimators. I can use R to take the sample x (a vector of 25 observations) and use the sample to find all three estimators:

set.seed(1022)
x = rnorm(25, 100, 15)
summary(x)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
64.26   88.07   99.40  102.28  114.77  140.97


Then we see that $$A = 102.28,$$ $$H = 99.40,$$ $$D = (64.26+140.97)/2 = 102.615.$$ So all three estimators of $$\mu$$ give values close to $$100;$$ just from this brief experiment, it seems that all three estimators may be reasonably good in practice.

Above, I claimed that all three estimators are unbiased. A formal mathematical proof of unbiasedness may be above your mathematical level. But your text says that the expected value of an estimator may be obtained by taking the average value of all possible samples of a given size (here 25) drawn from the population.

Obviously, it is not possible to look at "all possible samples" of size 25, but we can use R to take a huge number (say 100,000) of samples, find $$A, H,$$ and $$D$$ for each sample, and average the results. That will give a good approximation of the three expected values.

One way to do this in R is shown below.

set.seed(2020)
m = 10^5;  n = 25
DTA = matrix(rnorm(m*n, 100, 15), nrow=m)


DTA is a matrix with m rows and n columns; each row is a sample of size $$n = 25.$$ Then we can find $$m = 100000$$ values each of the three estimators as follows.

a = rowMeans(DTA)
h = apply(DTA, 1, median)
mn = apply(DTA, 1, min);  mx = apply(DTA, 1, max)
d = (mn+mx)/2
mean(a);  mean(h);  mean(d)
[1] 99.99417   # aprx E(A) = 100
[1] 99.99105   # aprx E(H) = 100
[1] 100.0083   # aprx E(D) = 100


By taking the averages of 100,000 samples (instead of "all possible" samples), we have reasonable evidence that all three estimators are unbiased.

However, not all three estimators $$A, H,$$ and $$D$$ are equally useful in practice. We would like for an estimator to give a value near to the parameter (here $$\mu)$$ being estimated as possible.

Among unbiased estimators it is customary to use the one with the smallest variance. Again using our 100,000 samples of size $$n = 25,$$ we can get a good idea which of the three estimators to use. The winner here is the sample mean $$A = \bar X.$$ Later you may see the formula that $$Var(\bar X) = \sigma^2/n,$$ which is $$15^2/25 = 9$$ in our example.

var(a);  var(h);  var(d)
[1] 8.926405  # aprx Var(A) = 9
[1] 13.84278  # aprx Var(H) > 9
[1] 29.84585  # aprx Var(D) > 9


======================

Methods of calculus are required to find the population mean $$\mu$$ from the density function of a normal distribution. Below are details for finding the population mean from the PDF of a binomial distribution.

If $$X \sim \mathsf{Binom}(n=3, p=1/2),$$ then $$E(X) = \sum_{k=0}^3 kP(X=k) = \sum_{k=0}^3 k\cdot{3\choose k}(1/2)^3\\ = 0(1/8) + 1(3/8) + 2(3/8) + 3(1/8) = 12/8 = 1.5.$$

Let's put it into the format of a table:

k       P(X=k)      kP(X=k)
---------------------------
0         1/8         0
1         3/8        3/8
2         3/8        6/8
3         1/8        3/8
----------------------------
Totals:    1         1.5 = E(X)


set.seed(2020)
x = rbinom(10^6, 3, .5)
table(x)/10^5
x
0       1       2       3
1.24520 3.75870 3.74821 1.24789
mean(x)
[1] 1.499879


In the figure below the blue bars show simulated values of $$X$$ and red dots show the PDF of $$X.$$ They match closely enough that it is difficult to see any differences at the resolution of the plot.

hdr = "Million Realizations of BINOM(3,.5) with PDF"
hist(x, br=seq(-.5,3.5), prob=T, col="skyblue2", main=hdr)
k = 0:3;  pdf = dbinom(k, 3, .5)
points(k, pdf, col="red", pch=19)


• Found 6th edition from 25 yrs ago. Same topic still in Ch6, Sect 1. This edition does not show how to find expectation of a random variable, but states sample mean $\bar X = \frac{\sum X}{n}$ may be used to estimate population mean $\mu.$ Also, says sample median might be used. Then talks about how to decide which is better. Mentions unbiasedness as criterion. Can you carefully quote sentence where you start to get lost. Also, what was last topic discussed before starting Ch 6? // Finally, please delete some of your earlier Comments. Oct 22, 2020 at 23:29
• I don't understand this sentence mentioned above : "𝐸(𝑇) is obtained by taking the average value of 𝑇 computed from all possible samples of a given size that may be drawn from the population. That is, 𝐸(𝑇)=𝜇𝑇".. // The last topic was the sampling distributions... @BruceET Oct 22, 2020 at 23:34
• I know that thank you comments are not encouraged, but I really want to thank you for taking the time to answer my question, so thanks a lot! @BruceET Oct 23, 2020 at 9:26