14
$\begingroup$

I have read different descriptions of censored data:

A) As explained in this thread, unquantified data below or above a certain threshold is censored. Unquantified means data is above or below a certain threshold but we do not know the exact value. Data is then marked at the low or high threshold value in the regression model. It matches the description in this presentation, which I've found very clear (2nd slide on first page). In other words $Y$ is capped to either a minimum, a maximum value or both because we do not know the true value outside of that range.

B) A friend told me that we can apply a censored data model to partially unknown $Y$ observations, provided we have at least some limit information about the unknown $Y_i$ outcomes. For example, we want to estimate the final price for a mix of silent and open auctions based on some qualitative criteria (type of goods, country, bidders wealth, etc.). While for the open auctions we know all final prices $Y_i$, for the silent auctions we only know the first bid (say, $1,000) but not the final price. I was told that in this case data is censored from above and a censored regression model should be applied.

C) Finally there is the definition given by the Wikipedia where $Y$ is missing altogether but the predictors are available. I'm not sure how this example is different from truncated data.

So what exactly are censored data?

$\endgroup$
  • 6
    $\begingroup$ The more relevant Wikipedia article is at en.wikipedia.org/wiki/Censoring_%28statistics%29. Although not comprehensive, it at least describes Type I and Type II censoring and acknowledges interval censoring along with left and right censoring. $\endgroup$ – whuber Oct 21 '14 at 19:19
8
+50
$\begingroup$

Consider the following data on an outcome $y$ and a covariate $x$:

user y       x   
1    10      2 
2   (-∞,5]   3 
3   [4,+∞)   5   
4   [8,9]    7
5     .      .

For user 1, we have the complete data. For everyone else, we have incomplete data. Users 2, 3 and 4 are all censored: the outcome corresponding to known values of the covariate is not observed or is not observed exactly (left-, right-, and interval-censored). Sometimes this is an artifact of privacy considerations in survey design. In other times, it happens for other reasons. For instance, we don't observe any wages below the minimum wages or the actual demand for concert tickets above the arena capacity.

User 5 is truncated: both the outcome and the covariate are missing. This usually happens because we only collect data on people who did something. For instance, we only survey people who bought something ($y>0$), so we exclude anyone with $y=0$ along with their $x$s. We may not even have a row for this type of user in out data, though we know they exist because we know the rule that was used to generate our sample. Another example is incidental truncation: we only observe wage offers for people who are in the work force, because we assume that the wage offer is the wage when you are working. The truncation is incidental since it depends not on $y$, but on another variable.

In short, truncation implies a greater information loss than censoring (points A & B). Both of these types of "missingness" are systematic.

Working with this type of data typically involves making a strong distribution assumption about the error, and modifying the likelihood to take this into account. More flexible semi-parametric approaches are also possible. This is implicit in your point B.

$\endgroup$
  • 2
    $\begingroup$ Two aspects of this answer confuse me. First, purely missing values do not necessarily reflect truncation. Second, the manner of censoring ("random" vs. "informative") is often as important as the fact of censoring, indicating that there is more to censoring than the mere recording of interval-valued data. $\endgroup$ – whuber Oct 21 '14 at 21:31
  • $\begingroup$ This is a very good example. Does it mean that different "censoring limits" can be applied to each $y$? How do we formulate the model in this case? This would validate my friend statement in B). $\endgroup$ – Robert Kubrick Oct 21 '14 at 21:34
  • 1
    $\begingroup$ If you're willing to assume homoskedastic, normally-distributed errors, the likelihood can be written like this and you can use MLE with observation-specific censoring thresholds. $\endgroup$ – Dimitriy V. Masterov Oct 21 '14 at 21:53
  • 3
    $\begingroup$ @Peter Isn't that (right) censoring? Truncation would occur when all such people are excluded from and not counted in the sample at all. See en.wikipedia.org/wiki/Truncation_(statistics). $\endgroup$ – whuber Apr 24 '18 at 22:35
  • 1
    $\begingroup$ @Peter What is confusing is you write "then we do know that that person's BMI is over 30": how can you possibly refer to somebody who isn't even in your sample ("didn't get counted")? An observation of the form "BMI > 30" is censoring whereas the complete exclusion of all such observations from your analysis, even when they might exist in the population, is truncation. In the latter case all you can say is "people with BMI above 30 have been excluded from the sample." $\endgroup$ – whuber Apr 25 '18 at 13:10
9
$\begingroup$

Descriptively speaking, I would offer "a data sample is censored if some observations in it take on, or constitute, the sample's extreme values but their true value is outside the observed sample range". But this is deceptively straightforward.

So let's first discuss how we can conclude that a data set is censored, which will naturally lead us to discussing the cases presented in the question.

Suppose we are given the following data set from a discrete random variable $X$, for which the only thing we know is that it is non-negative:

$$\{0,1,1,2,2,2,2,2,2,2\}$$

Can we say that the data set is censored? Well, we are entitled to think that it might be, but it is not necessarily so:

1) $X$ may have the range $\{0,1,2\}$ and a probability distribution $\{0.1,0.1,0.8\}$. If this is indeed the case, it appears there is no censoring here, just an "anticipated" sample from such a random variable, with bounded support and highly asymmetrical distribution.

2) But it may be the case that $X$ has the range $\{0,1,...,9\}$ with uniform probability distribution $\{0.1,0.1,...0.1\}$, in which case our data sample is most likely censored.

How can we tell? We cannot, except if we posses prior knowledge or information, that will permit us to argue in favor of the one or the other case. Do the three cases presented in the question represent prior knowledge to the effect of censoring? Let's see:

Case A) describes a situation where for some observations we have only qualitative information like "very large", "very small" etc, which leads us to assign to the observation an extreme value. Note that merely not knowing the actual realized value does not justify assigning an extreme value. So we must have some information to the effect that for these observations, their value exceed or is below all the observed ones. In this case, the actual range of the random variable is unknown, but our qualitative information permits us to create a censored sample (it is another discussion as to why we do not just drop the observations for which we do not possess the actual realized value).

Case B) is not a case of censoring, if I understand it correctly, but rather a case of contaminated sample: our a priori information tells us that the maximum value of the random variable cannot exceed $3$ (due say to a physical law or a social law -suppose this is grades data from a grading system that uses only the values $1,2,3$). But we have observed also the value $4$ and the value $5$. How can this be? Mistake in the recording of the data. But in such a case, we do not know for certain that the $4$'s and $5$'s should be all $3$'s (in fact, looking at the side keyboard of a computer, it is more likely that the $4$'s are $1$'s and the $5$'s are $2$'s!). By "correcting" in whatever way the sample, we do not make it a censored one, because the random variable is not supposed to range in the recorded range in the first place (so there are no true probabilities assigned to the values $4$ and $5$).

Case C) refers to a joint sample, where we have a dependent variable and predictors. Here, we may have a sample where values of the dependent variable are concentrated at the one or both extremes, due to the structure of the phenomenon under study: In the "hours worked" usual example, unemployed people do not work but they would have worked (think carefully: does this case really falls under the descriptive "definition" in the beginning of this answer?). So including them in the regression with recorded hours "zero" create bias. To the other extreme, maximum numbers of hour worked may be argued to be able to reach, say $16$/day, and there may be employees that would be willing to work so many for given pay. But the legal framework does not permit it and so we do not observe such "hours worked". Here, we are trying to estimate the "intended labour supply function" -and it is with respect to this variable that the sample is characterized as censored.
But if we declared that what we want to do is to estimate "labor supply function given the phenomenon of unemployment and the legal framework", the sample would not be censored, since it would reflect the effect of these two aspects, something that we want it to do.

So we see that characterizing a data sample as censored
a) can come from different situations and
b) requires some care
-let alone the fact that it can be confused with the case of truncation.

$\endgroup$
  • 6
    $\begingroup$ This seems to be an econometric perspective. Note that it is common in biomedical research to have durations (possibly literally survival) as a response, & to have patients censored by not having experienced the event by the end of the observation period. But also to have patients who dropped out or were lost to follow up during the observation period. (Perhaps they moved away & contact was lost.) We can know that the survival time is > the last contact, but could be shorter than the end of the observation period. $\endgroup$ – gung - Reinstate Monica Oct 21 '14 at 17:28
  • $\begingroup$ Case B is not for wrong or contaminated data. Suppose we want to estimate the final price of a mix of silent and open auctions based on some qualitative criteria (type of goods, country, bidders wealth, ...). For the silent auctions we only know the first bid (say, $1,000) but not the final price. I was told we can use the silent auctions data by using some form of censored modeling. $\endgroup$ – Robert Kubrick Oct 21 '14 at 19:30
  • 1
    $\begingroup$ @gung this is certainly an econometric approach, given who wrote the answer! $\endgroup$ – Alecos Papadopoulos Oct 21 '14 at 20:11
  • 1
    $\begingroup$ @RobertKunrick What you describe does not match to case B. The way Case B is described, we observe a range of values, and then we are told that some of the observed values are in reality impossible. How does that match with the auctions example? $\endgroup$ – Alecos Papadopoulos Oct 21 '14 at 20:13
  • $\begingroup$ Please don't take it as a criticism, @AlecosPapadopoulos. I don't think there's anything incorrect. I just want to point out that terms are used differently in different fields, & this isn't the convention is stat / biostat. $\endgroup$ – gung - Reinstate Monica Oct 21 '14 at 20:15
2
$\begingroup$

For me, censoring means that we observe partial-information about an observation $Z_i$. What I mean by this is that, rather than observing $Z_i = z_i$ we observe $Z_i \in a_i$ where $a_i$ is the realization of $A_i$, which is some random coarsening of the sample space. We might imagine that we first select a partition $\mathcal A_i$ of the sample space $\mathcal Z$, then $Z_i$ is generated, and we report the $A_i \in \mathcal A_i$ such that $Z_i \in A_i$ (equivalently, we report $I(Z_i \in A)$ for all $A \in \mathcal A_i$). Uninformative censoring of $Z_i$, for example, then means that $\mathcal A_i$ is independent of $Z_i$.

This is a little heuristic and sloppy. We should probably also require that the distribution of $[Z_i \mid Z_i \in a_i]$ is non-degenerate to consider $Z_i$ censored. We also might note that, as defined, this is a generalization of missing data where for $Z_i = (X_i, Y_i)$ one might say $Y_i$ is missing if $a_i = \{x\} \times \mathcal Y$ where $\mathcal Y$ is the sample space of $Y$ and say $Z_i$ is missing if $a_i = \mathcal Z$. When one says "$Z_i$ is censored", if they are following my definition, what they usually mean is "$Z_i$ is censored, but is not missing".

$\endgroup$
1
$\begingroup$

It's important to distinguish censored versus truncated as well as missing data.

Censoring applies specifically to the issue of survival analysis and time-to-event outcomes wherein the event at hand is assumed to have occurred at some time past the point at which you stopped observing that individual. An example is men-who-have-sex-with-men (MSM) and the risk of incident HIV in a prospective study who move and cease contact with study coordinators.

Truncation applies to a continuous variable that evaluates to a specific point at which the actual value is known to be either greater than or less than that point. An example is the monitoring of subjects with HIV and the development of full blown AIDS, CD4 cell counts falling below 300 are evaluated to the lower-limit-of-detection 300.

Lastly, missing data are data that have actual values that are not observed in any sense. Censored data are not missing time-to-event data nor are they truncated.

$\endgroup$
  • 1
    $\begingroup$ There's another use of "truncation": to describe a data-generating process where observations above/below cut-offs are unobtainable. A classic example involves counting the no.eggs found in the nests of a particular bird species, where the species can only be identified from the egg; empty nests could be from any species so the no. zeroes is unknown. If the no. eggs follows a Poisson distribution, the egg counts from non-empty nests follow a truncated Poisson. So truncation produces missing data according to a specific well defined mechanism. $\endgroup$ – Scortchi - Reinstate Monica Oct 22 '14 at 14:40
  • 1
    $\begingroup$ ... Your cell count data are indeed censored according to many people's understanding of the term, which is not restricted to time-to-event measurements, because you know everything about each subject except how far below 300 his cell count is; "truncation" here (alternatively "Winsorization") describes the method of analysis, viz the treatment of values below 300 as if they were equal to 300. $\endgroup$ – Scortchi - Reinstate Monica Oct 22 '14 at 14:41
  • $\begingroup$ A clear reference on the survival analysis concept of censoring: itl.nist.gov/div898/handbook/apr/section1/apr131.htm. $\endgroup$ – Eric O Lebigot Aug 8 '16 at 9:54
-1
$\begingroup$
  1. Censored: This is a term used to indicate that the period of observation was cut off before the event of interest occurred. So ''censored data'' indicate that the period of a particular event as not or never occurred
$\endgroup$
  • 3
    $\begingroup$ Welcome to the site. If this is copied from some source, please cite the source. $\endgroup$ – gung - Reinstate Monica Oct 21 '14 at 19:30
  • 3
    $\begingroup$ Censoring applies to much more than time-dependent observations. For instance, chemical concentrations measurements that are below the limit of detection are censored, too. $\endgroup$ – whuber Oct 21 '14 at 21:33
  • $\begingroup$ @whuber: May I offer a friendly amendment to that observation. The chemical concentrations that fall below the limit of detection are indeed censored, but since they cannot possibly be negative, analyses should consider them as truncated at zero. My own understanding of the truncation-censoring distinction is that truncation applies to the range of parameters that are possible for the underlying distribution. $\endgroup$ – DWin Jul 10 at 15:57
  • $\begingroup$ @DWin Thank you for that thoughtful clarification. I can only agree with the first point. In the great majority of datasets I have analyzed, though, it was necessary to re-express concentrations as logarithms--and there the distinction disappears. In other datasets where background has been subtracted (such as radiological measurements), there is no definite left endpoint, either. Your second point strikes me as unusual: I have never seen "truncation" used to refer to creating a subset of a distribution family. $\endgroup$ – whuber Jul 10 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.