Inference for a distribution with different number of samples due to censoring

Question

I have the following problem in case someone has an idea about how to solve this.

Assume three experiments that refer to the same population for a random variable $X$.

In the first experiment, I observe samples in $x=\{1,2,3\}$ (frequencies $n_i^1$, $i=\{1,2,3\}$) but no higher values due to (right) censoring. In the second experiment, I observe samples in $x=\{1,2,3,4,5\}$ (frequencies $n_i^2$), and, finally, in the third one I observe samples in $x=\{1,\ldots,10\}$ (frequencies $n_i^3$).

I would like to obtain the empirical distribution function of the population in $\{1,\ldots, 10\}$ by making use of all the information. What would be a good way of combining the frequencies from all the experiments?

Answer 1

If I have understood this correctly, you are sampling on three occasions from the same population, defined by a distribution on the naturals, a count distribution. Let that distribution be defined by $P(X=i)=\pi_i, i=0,0,2,\dots$.

The counts n the first occasion are $N_{1j}, j=1,2,3$, where $N_{11}$ is the number of ones observed, and son on, while the last count $N_{13}$ counts three or larger. In the same way define the counts on the second and third occasion, where the last count inclueds "... or larger". The we can write the loglikelihood function as $$ \ell = (N_{11}+N_{21}+N_{31}) \log \pi_1 + (N_{12}+N_{22}+N_{32}) \log \pi_2 + (N_{23}+N_{33}) \log \pi_3 + \dots + N_{13} \log(1-\pi_1-\pi_2) + N_{25} \log(1-\pi_1-\pi_2-\pi_3-\pi_4) + N_{3,10}\log(1-\pi_1-\dots -\pi_9) $$ And then this loglikelihoodfunction should be maximized in the unknown parameters. This can be done non-parametrically as written above, or with some parametric model for the $\pi_i$'s. Continuing with the non-parametric case, first we find the partial derivative with respect to $\pi_1$, which is $$ \frac{\partial \ell}{\partial \pi_1} =\frac{N_{11}+N_{21}+N_{31}}{\pi_1} - \frac{N_{13}}{1-\pi_1-\pi_2} - \frac{N_{25}}{1-\pi_1-\dots -\pi_4} - \frac{N_{3,10}}{1-\pi_1-\dots -\pi_9} $$ then other partial derivatives can be calculated in like manner, and the equations solved. I leave that step as an exercise.

Answer 2

This is exactly the case that the Kaplan Meier curves are created from: in fact, they are a generalization of the Empirical Distribution Function that allows right censoring.

If you are familiar with R, they can be fit using the survfit function in the survival package.