Extrapolating the amount of data missing from the amount of data partially missing

Question

Suppose I have a fleet of two-car trains riding around, and that each car is equipped with a data recording device. Unfortunately, some of the recording devices aren't working. I don't know either the exact size of the fleet or the percentage of failed recording devices. I'd like to make a reasonable guess about how many cars I'm missing data from.

In particular:

• Total fleet size $S$ is unknown.
• Failure rate (failed units / total cars) $F$ is unknown.
• I have data from both cars from $S_2$ trains and from exactly one car from $S_1$ trains.
• Therefore, I know that there are at least $S_1$ failed units. What I don't know is $S_0$, the number of trains with failed units on both cars.
• Let's assume that the distribution of cars with failed units is random, and whether one car has a failed unit is independent of whether it's mate does.

Does the following make sense for a first-order approximation?

• Guess that the failure rate $F'$ is equal to the proportion of missing cars that I know about to total cars that I know about: $F' = S_1 / (2*(S_2 + S_1))$
• Assume that the likelihood of a train having two failed units is $F'^2$.
• Therefore $S_0 = F'^2 * (S_0 + S_1 + S_2)$
• Therefore $S_0 = (S_2 + S_1) * F'^2 / (1 - F'^2)$

Answer 1

This is a quick partial response to outline some options and correct some errors.

You are implicitly seeking a method of moments estimator. Under your assumptions, letting $f$ be the failure rate and $n$ be the fleet size, the expectations of the $S_i$ (which are governed by a multinomial distribution) are

$$\eqalign{ \mathbb{E}_{f;n}[S_0] = &f^2 n \cr \mathbb{E}_{f;n}[S_1] = &2 f (1-f) n \cr \mathbb{E}_{f;n}[S_2] = &(1-f)^2 n. }$$

From the algebraic relation

$$\mathbb{E}_{f;n}[S_0] = \frac{\mathbb{E}_{f;n}[S_1]^2}{4 \mathbb{E}_{f;n}[S_2]}$$

(which implies the failure rate should be about twice your estimate) and assuming $S_2 \ne 0$ you can derive the method of moments estimator

$$\hat{S}_0 = \frac{S_1^2}{4 S_2}.$$

This is likely to be biased, especially if $S_2$ is small, which impels us to consider other estimators.

More generally, this problem can be thought of as looking for a "good" estimator for $S_0$ of the form $\hat{S}_0 = t(S_1, S_2)$ in an experiment in which the outcomes are the sum of $n$ Binomial($1-f$, $2$) variables and $S_1$ and $S_2$ are the counts of the single and double "successes," respectively. To this end you need to supply a loss function $\Lambda(s,t)$ and analyze the statistical risk $r$, which is the expected loss

$$r_t(f,n) = \mathbb{E}_{f;n}[\Lambda(S_0, t(S_1, S_2))].$$

The loss function quantifies the cost of estimating that $S_0$ equals $t(S_1, S_2)$; usually the estimate is not perfect and there is a cost associated with that. The risk for a particular estimator $t$ is the expected loss; it depends on the unknown failure rate $f$ and the unknown fleet size $n$. Thus the exercise comes down to finding procedures with acceptable risk functions. If you have some quantitative information about the likely values of $f$ and $n$ you can exploit that either directly, by limiting the domain of the risk to the likely values, or with a Bayesian analysis in which you compute the expected value of $r_t$ under some assumed prior distribution of $(f,n)$. At this point the risk becomes solely a function of the procedure $t$ and it's "merely" a matter of finding a risk-minimizing procedure.

In any event, to make further progress you need to supply a loss function (or some reasonable approximation thereof). I would hesitate to recommend or use the method of moments estimator (derived above) without knowing something about your loss.