The Horvitz-Thompson Estimator is usually given by:

$$ \hat{Y}_{HT} = \sum_{i=1}^n \pi_i ^{-1} Y_i $$

The proof that it is unbiased is trivial to do. In additional, there exists other estimators out there for different designs as well, like those in Rubin and Rosenbaum (1983). However, in each of the original papers, the estimator seemed to appear out of nowhere with no motivation, only appearing so that the author could show it was unbiased.

My question is, is there a solid way to come up with unbiased estimators like there?

  • $\begingroup$ What do you mean by "usually"? That IS the definition. That this estimator is being applied to observational studies in matching tasks and the like is a bit unfortunate: H-T is proposed as a way to work with samples from finite population, while the use of weighted estimators in observational studies / matched samples is essentially to generalize to the "population" of size 2 per sample unit -- namely the two counterfactual outcomes. Variance of H-T is derived from the sampling principles; Rubin's book on matched sampling does not even have an index entry for variance. $\endgroup$ – StasK Sep 20 '17 at 23:50
  • $\begingroup$ Show us the "trivial" work on demonstrating unbiasedness. A posted answer also said this is "trivial". Well, maybe we still need to discuss those trivialities. And my personal favorites on sampling designs are Brewer & Hanif (1983) and Tille (2006); let's discuss these as well. $\endgroup$ – StasK Sep 20 '17 at 23:53
  • 1
    $\begingroup$ My question is, what went through the mind of authors in creating estimators like these? Was there a way to derive it or was it more on trial and error until unbiasedness was achieved? $\endgroup$ – user321627 Sep 21 '17 at 1:14
  • $\begingroup$ If you have a variable $I_i$ that takes values 0 and 1 with known probabilities, you know that the expected value is $\pi_i$, so if you want to make it look like 1, you need to divide by $\pi_i$. That's all. $\endgroup$ – StasK Sep 22 '17 at 1:25

the estimator seemed to appear out of nowhere with no motivation

If you think the idea behind stratified sampling is intuitive, then I believe Horvitz-Thompson should come as a natural extension, it's not something out of the blue.

To illustrate how a simple stratified sample could help you come up with the formula, consider a case with two strata, $S_1$ and $S_2$ of known sizes $N_1$ and $N_2$ and suppose you get samples $n_1$ and $n_2$ respectively. Now imagine you compute the average for each sample $\bar{y}_1$ and $\bar{y}_2$.

How would you use this information to estimate the total $Y$? The natural way is to take the estimated average of each stratum and multiply by the total (population) number of elements of the stratum:

$$ \hat{Y} = N_1 \bar{y}_1 + N_2 \bar{y}_2 $$

But take this simple expression and rewrite it as:

$$ \begin{align} \hat{Y} &= N_1 \sum_{i \in S_1}\frac{y_i}{n_1} + N_2 \sum_{i \in S_2}\frac{y_i}{n_2}\\ &= \sum_{i \in S_1}\frac{y_i}{n_1/N_1} + \sum_{i \in S_2}\frac{y_i}{n_2/N_2}\\ &= \sum_{i} \frac{y_i}{\pi_i} \end{align} $$

Where $\pi_i = n_1/N_1$ if $i\in S_1$ and $\pi_i = n_2/N_2$ if $i\in S_2$. That is, a simple stratified sampling already gives you the insight that, in essence, what we are doing is summing each sampled $y_i$ upweighted by its probability of selection. Then the idea of a general inverse probability weighting, where each $\pi_i$ could be different, should come naturally.

  • $\begingroup$ I don't see any single expectation. How can you claim unbiasedness if you have not taken an expected value?? $\endgroup$ – StasK Sep 22 '17 at 1:25
  • 1
    $\begingroup$ @StasK the OP is not asking about unbiasedness, he already knows it's unbiased (and the proof is trivial). The OP asked about how could anybody have the intuition to come up with the formula. And indeed, some texts present the formula without any hint or context. $\endgroup$ – Carlos Cinelli Sep 22 '17 at 2:20

I'd like to take a different approach to the accepted answer. The accepted answer justifies the intuition behind the inverse probability weighting, but I'd like to justify why the Horvitz-Thompson Estimator makes sense to arrive at rather than some other estimator. It can stem from two properties:

  1. The estimator is a linear combination of survey responses. This property is an advantage because it allows us to calculate expectations (and variances) without considering all possible samples, and instead only deal with individual units' (joint) probabilities of selection.

  2. The estimator is unbiased (as you already note). This is a nice property because it makes the eventual estimate easier to interpret/explain to someone else.

The Horvitz-Thompson Estimator is the unique estimator with these two properties. In my opinion, the most intuitive way to approach it is to:

  1. Justify the two properties above,

  2. Use property 1 to define the structure of the estimator ($\sum w_i y_i$),

  3. Use property 2 to show that $w_i=\pi_i^{-1}$, and then,

  4. Show that this is unique (e.g. consider particular $y_i$ values)

  • $\begingroup$ That's how justification of Horvitz-Thompson estimator is offered in sampling books. I would note though that the random quantities are sample inclusion indicators, and unbiasedness of Horvitz-Thompson estimator is with respect to these (with $y_i$ treated as fixed). $\endgroup$ – StasK Sep 22 '17 at 22:47
  • $\begingroup$ Maybe I'm too close to the subject. This is the intuitive answer for me. When I had "Why this?" questions, it was property 1's justification that cleared them up. $\endgroup$ – RoryT Sep 23 '17 at 0:10
  • $\begingroup$ @StasK formal justification is not how one would come up with the idea. For example, it's not like newton invented calculus thinking about the proper definition of limits etc. RoryT your answer is perfectly fine too! One question though -- say HT did not exist. Would you say you would naturally intuit about it in this way and invent it yourself? $\endgroup$ – Carlos Cinelli Sep 23 '17 at 16:42

First, note that the Horvitz-Thompson (H-T) estimator is used for both $\textit{with or without replacement}$ random sampling designs.

$\pi$ is the inclusion probability. The H-T estimator provides us with an estimate of the population total.

As a simple illustration, a Horvitz-Thompson "like" estimator can be derived as follows:

If we let $\pi = \frac{n}{N}$, where $n$ is the sample size and $N$ is the population size, and substitute into the H-T estimator, then, after simplification, we get that $\hat{Y} = N\frac{Y}{n} = N\bar{y}$.

Showing that such an estimator is unbiased is now a straightforward task.

  • 1
    $\begingroup$ You don't understand what the probability space here is. It is that of sampling indicators, not of measurements $y_i$. $\endgroup$ – StasK Sep 20 '17 at 23:48
  • 1
    $\begingroup$ @StasK You're absolutely correct on this! Thank you for pointing it out! I just noticed my major flaw in writing that $\pi = N\bar{y}$ which is not true and can be easily misconstrued. I'd also like to add that the estimator that I gave above can be viewed as a "Horvitz-Thompson like" estimator. I'll make the correction. $\endgroup$ – compbiostats Sep 21 '17 at 0:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.