8
$\begingroup$

I am having a little trouble understanding the concept and derivation of the likelihood of truncated data.

For example, if I want to find the likelihood function based on a sample from a distribution, but when taking a sample from the distribution, I observe the truncated values (where there is a cut-off of $M$, i.e. any $x_{i}>M$ is recorded as $M$):

$ x_{1}, x_{2}, M, x_{3}, M, x_{4}, x_{5}, ..., x_{10}$

where the number of $M$ values is $m$. Then, the likelihood is supposedly given by:

$L(x;\theta) = \prod_{i=1}^{10}f(x_{i};\theta)*[P(X>M)]^{m}$

I would very much appreciate an explanation/proof of why this is so, importantly why the second factor is as it is. Intuitively and mathematically if possible. Thanks very much in advance.

$\endgroup$
  • $\begingroup$ What is lower-case "$m$"? $\endgroup$ – Alecos Papadopoulos Nov 28 '13 at 1:47
  • $\begingroup$ It is the number of occurences of $M$.. i.e. I have observed $10 + m$ data points, of which $10$ are not truncated, and $m$ of them are (I observe these $m$ picks, all with value $M$) $\endgroup$ – Delvesy Nov 28 '13 at 11:51
  • 2
    $\begingroup$ As @Alecos points out, you're using "truncated" idiosyncratically. "Censored" is the usual term. $\endgroup$ – Scortchi - Reinstate Monica Nov 29 '13 at 0:21
  • $\begingroup$ Some other terms you may want to search on: "ceiling/floor effects", "beta regression", and "zero-inflated models." $\endgroup$ – DWin Dec 2 '13 at 0:00
8
$\begingroup$

What you describe needs special treatment, it is not what we usually mean by "truncated random variables"-and what we usually mean is that the random variable does not range outside the truncated support, meaning that there is not a concentration of probability mass at the point of truncation. To contrast cases:

A) "Usual" meaning of a truncated rv
For any distribution that we truncate its support, we must "correct" its density so that it integrates to unity when integrated over the truncated support. If the variable has support in $[a,b]$, $-\infty < a < b < \infty$, then (pdf $f$, cdf $F$)

$$\int_a^bf_X(x)dx = \int_a^Mf_X(x)dx+\int_M^bf_X(x)dx = \int_a^Mf_X(x)dx + \left[1-F_X(M)\right]=1 $$

$$\Rightarrow \int_a^Mf_X(x)dx = F_X(M)$$

Since the LHS is the integral over the truncated support, we see that the density of the truncated r.v., call it $\tilde X$, must be

$$f_{\tilde X}(\tilde x) = f_{X}(x\mid X\le M)=f_X(x)dx\cdot \left[F_X(M)\right]^{-1} $$ so that it integrates to unity over $[a, M]$. The middle term in the above expression makes us think of this situation (rightfully) as a form of conditioning -but not on another random variable, but on the possible values the r.v. itself can take. Here a joint density/likelihood function of a collection of $n$ truncated i.i.d r.v.'s would be $n$ times the above density, as usual.

B) Probability mass concentration
Here, which is what you describe in the question, things are different. The point $M$ concentrates all the probability mass that corresponds to the support of the variable higher than $M$. This creates a point of discontinuity in the density and makes it having two branches

$$\begin{align} f_{X^*}(x^*) &= f_X(x^*) \qquad x^*<M\\ f_{X^*}(x^*) &= P(X^* \ge M) \qquad x^*\ge M\\ \end{align}$$

Informally, the second is "like a discrete r.v." where each point in the probability mass function represents actual probabilities. Now assume that we have $n$ such i.i.d random variables, and we want to form their joint density/likelihood function. Before looking at the actual sample, what branch should we choose? We cannot make that decision so we have to somehow include both. To do this we need to use indicator functions: denote $I\{x^*\ge M\}\equiv I_{\ge M}(x^*)$ the indicator function that takes the value $1$ when $x^*\ge M$, and $0$ otherwise. The density of such a r.v. can be written

$$f_{X^*}(x^*) = f_X(x^*)\cdot \left[1-I_{\ge M}(x^*)\right]+P(X^* \ge M)\cdot I_{\ge M}(x^*) $$ and therefore the joint density function of $n$ such i.i.d. variables is

$$f_{X^*}(\mathbf X^*\mid \theta) = \prod_{i=1}^n\Big[f_X(x^*_i)\cdot \left[1-I_{\ge M}(x^*_i)\right]+P(X^*_i \ge M)\cdot I_{\ge M}(x^*_i)\Big]$$

Now, the above viewed as a likelihood function, the actual sample consisting of realizations of these $n$ random variables comes into play. And in this sample, some observed realizations will be lower than the threshold $M$, some equal. Denote $m$ the number of realizations in the sample that equals $M$, and $v$ all the rest, $m+v=n$. It is immediate that for the $m$ realizations, the corresponding part of the density that will remain in the likelihood will be the $P(X^*_i \ge M)$ part, while for the $v$ realizations, the other part. Then

$$\begin{align} L(\theta\mid \{x_i^*;\,i=1,...n\})&= \prod_{i=1}^v\Big[f_X(x^*_i)\Big]\cdot \prod_{j=1}^m\Big[P(X^*_j \ge M)\Big] \\& = \prod_{i=1}^v\Big[f_X(x^*_i)\Big]\cdot \Big[P(X^* \ge M)\Big]^m\\ \end{align}$$

$\endgroup$
  • $\begingroup$ Thank you. I very much appreciate the reply. I guess my main issue is the first point under section b)... i.e., how the "second branch" of the pdf is defined. It is a discrete pmf and doesn't really define a pdf from the definition of a pdf. Could this section be explained further? Thanks very much. $\endgroup$ – Delvesy Nov 28 '13 at 21:52
  • $\begingroup$ These random variables are called "mixed-type", i.e. they are partly continuous and partly discrete. Intuitively it makes obvious sense, as your questions shows. For a rigorous treatment, look up "mixed type random variables" or "mixed type distributions". DON'T confuse them with "mixtures". $\endgroup$ – Alecos Papadopoulos Nov 28 '13 at 22:40
4
$\begingroup$

The likelihood theory is a fairly general framework. Most textbooks state results for the separated cases of continuous r.vs and for that for discrete r.vs. However mixed cases occur in practice, as is the case here.

For a discrete r.v. $A$, the likelihood of an observation $a$ is defined as the probability of getting the observed value $a$, say $p_A(a)$. For a continuous r.v. the likelihood $L$ is usually defined as the density at $x$, say $f_X(x)$. However in practice one only knows that $x_{\textrm{L}} < X < x_{\textrm{U}}$ - because of a limited measurement precision, and $\Pr\left\{x_{\textrm{L}} < X < x_{\textrm{U}}\right\}$ should be used as likelihood. Taking $x_{\textrm{L}}:= x - \textrm{d}x/2$, $x_{\textrm{U}}:= x + \textrm{d}x/2$ with $\mathrm{d}x$ small, we get $f_X(x)$ up to a multiplicative $\mathrm{d}x$ which does not matter. So the usual definition can be viewed as implicitly assuming an infinite precision on the observation.

For a couple of r.vs $A$ and $X$ with mixed joint type discrete/continuous, the likelihood will be the joint distribution, which is usually expressed using conditional distributions, e.g. $$ L := \textrm{Pr}\left\{ A = a, \, x_{\textrm{L}} < X < x_{\textrm{U}} \right\} = \textrm{Pr}\left\{ A = a \right\} \times \textrm{Pr} \left\{x_{\textrm{L}} < X < x_{\textrm{U}} \, \vert\, A = a\right\}. $$ Thus for an interval $(x_{\textrm{L}},\, x_{\textrm{U}})$ with small length $\textrm{d}x$, $L$ is $p_A(a)$ times the density of $X$ conditional on $\{A=a\}$, say $f_{X \vert A}(x \,\vert \,a)$. Again, we omit the $\mathrm{d}x$ term.

Now let us come back to your example, and consider only one observation. Then $A = 1_{\{X > M\}}$ is a Bernoulli r.v with success probability $\Pr\{X > M\}$. Depending on $X > M$ or not, either you observe only $A = 1$ or you observe both $A = 0$ and the value $x$ of $X$. In both cases you use the formula above, but $(x_{\textrm{L}},\, x_{\textrm{U}})$ is taken either as $(M,\,\infty)$ or as an interval of small length $\textrm{d}x$ containing $x$. Indeed this gives $$ L = \begin{cases} \textrm{Pr} \left\{X > M \right\} \times 1 & \textrm{if } X > M \textrm{ i.e. } A =1,\\ \textrm{Pr} \left\{X \leq M\right\} \times f_{X \vert A}(x \,\vert \,a)\,\textrm{d}x & \textrm{if } X \leq M \textrm{ i.e. } A = 0. \end{cases} $$ Since $f_{X \vert A}(x \,\vert \,0) = f_X(x) / \textrm{Pr} \left\{ X \leq M \right\}$, the likelihood is simply $f_X(x)\,\textrm{d}x$ in the second case and we get the claimed likelihood, up to the $\mathrm{d}x$ term for an observation with infinite precision. When independent observations $A_i$ and $X_i$ are made, the likelihood is obtained as the product of the marginal likelihoods leading to the expression in the question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.