What exactly are censored data?

Question

I have read different descriptions of censored data:

A) As explained in this thread, unquantified data below or above a certain threshold is censored. Unquantified means data is above or below a certain threshold but we do not know the exact value. Data is then marked at the low or high threshold value in the regression model. It matches the description in this presentation, which I've found very clear (2nd slide on first page). In other words $Y$ is capped to either a minimum, a maximum value or both because we do not know the true value outside of that range.

B) A friend told me that we can apply a censored data model to partially unknown $Y$ observations, provided we have at least some limit information about the unknown $Y_i$ outcomes. For example, we want to estimate the final price for a mix of silent and open auctions based on some qualitative criteria (type of goods, country, bidders wealth, etc.). While for the open auctions we know all final prices $Y_i$, for the silent auctions we only know the first bid (say, $1,000) but not the final price. I was told that in this case data is censored from above and a censored regression model should be applied.

C) Finally there is the definition given by the Wikipedia where $Y$ is missing altogether but the predictors are available. I'm not sure how this example is different from truncated data.

So what exactly are censored data?

Answer 1

Consider the following data on an outcome $y$ and a covariate $x$:

user y       x   
1    10      2 
2   (-∞,5]   3 
3   [4,+∞)   5   
4   [8,9]    7
5     .      .

For user 1, we have the complete data. For everyone else, we have incomplete data. Users 2, 3 and 4 are all censored: the outcome corresponding to known values of the covariate is not observed or is not observed exactly (left-, right-, and interval-censored). Sometimes this is an artifact of privacy considerations in survey design. In other times, it happens for other reasons. For instance, we don't observe any wages below the minimum wages or the actual demand for concert tickets above the arena capacity.

User 5 is truncated: both the outcome and the covariate are missing. This usually happens because we only collect data on people who did something. For instance, we only survey people who bought something ($y>0$), so we exclude anyone with $y=0$ along with their $x$s. We may not even have a row for this type of user in out data, though we know they exist because we know the rule that was used to generate our sample. Another example is incidental truncation: we only observe wage offers for people who are in the work force, because we assume that the wage offer is the wage when you are working. The truncation is incidental since it depends not on $y$, but on another variable.

In short, truncation implies a greater information loss than censoring (points A & B). Both of these types of "missingness" are systematic.

Working with this type of data typically involves making a strong distribution assumption about the error, and modifying the likelihood to take this into account. More flexible semi-parametric approaches are also possible. This is implicit in your point B.

Answer 2

Descriptively speaking, I would offer "a data sample is censored if some observations in it take on, or constitute, the sample's extreme values but their true value is outside the observed sample range". But this is deceptively straightforward.

So let's first discuss how we can conclude that a data set is censored, which will naturally lead us to discussing the cases presented in the question.

Suppose we are given the following data set from a discrete random variable $X$, for which the only thing we know is that it is non-negative:

$$\{0,1,1,2,2,2,2,2,2,2\}$$

Can we say that the data set is censored? Well, we are entitled to think that it might be, but it is not necessarily so:

1) $X$ may have the range $\{0,1,2\}$ and a probability distribution $\{0.1,0.1,0.8\}$. If this is indeed the case, it appears there is no censoring here, just an "anticipated" sample from such a random variable, with bounded support and highly asymmetrical distribution.

2) But it may be the case that $X$ has the range $\{0,1,...,9\}$ with uniform probability distribution $\{0.1,0.1,...0.1\}$, in which case our data sample is most likely censored.

How can we tell? We cannot, except if we posses prior knowledge or information, that will permit us to argue in favor of the one or the other case. Do the three cases presented in the question represent prior knowledge to the effect of censoring? Let's see:

Case A) describes a situation where for some observations we have only qualitative information like "very large", "very small" etc, which leads us to assign to the observation an extreme value. Note that merely not knowing the actual realized value does not justify assigning an extreme value. So we must have some information to the effect that for these observations, their value exceed or is below all the observed ones. In this case, the actual range of the random variable is unknown, but our qualitative information permits us to create a censored sample (it is another discussion as to why we do not just drop the observations for which we do not possess the actual realized value).

Case B) is not a case of censoring, if I understand it correctly, but rather a case of contaminated sample: our a priori information tells us that the maximum value of the random variable cannot exceed $3$ (due say to a physical law or a social law -suppose this is grades data from a grading system that uses only the values $1,2,3$). But we have observed also the value $4$ and the value $5$. How can this be? Mistake in the recording of the data. But in such a case, we do not know for certain that the $4$'s and $5$'s should be all $3$'s (in fact, looking at the side keyboard of a computer, it is more likely that the $4$'s are $1$'s and the $5$'s are $2$'s!). By "correcting" in whatever way the sample, we do not make it a censored one, because the random variable is not supposed to range in the recorded range in the first place (so there are no true probabilities assigned to the values $4$ and $5$).

Case C) refers to a joint sample, where we have a dependent variable and predictors. Here, we may have a sample where values of the dependent variable are concentrated at the one or both extremes, due to the structure of the phenomenon under study: In the "hours worked" usual example, unemployed people do not work but they would have worked (think carefully: does this case really falls under the descriptive "definition" in the beginning of this answer?). So including them in the regression with recorded hours "zero" create bias. To the other extreme, maximum numbers of hour worked may be argued to be able to reach, say $16$/day, and there may be employees that would be willing to work so many for given pay. But the legal framework does not permit it and so we do not observe such "hours worked". Here, we are trying to estimate the "intended labour supply function" -and it is with respect to this variable that the sample is characterized as censored.
But if we declared that what we want to do is to estimate "labor supply function given the phenomenon of unemployment and the legal framework", the sample would not be censored, since it would reflect the effect of these two aspects, something that we want it to do.

So we see that characterizing a data sample as censored
a) can come from different situations and
b) requires some care
-let alone the fact that it can be confused with the case of truncation.

Answer 3

For me, censoring means that we observe partial-information about an observation $Z_i$. What I mean by this is that, rather than observing $Z_i = z_i$ we observe $Z_i \in a_i$ where $a_i$ is the realization of $A_i$, which is some random coarsening of the sample space. We might imagine that we first select a partition $\mathcal A_i$ of the sample space $\mathcal Z$, then $Z_i$ is generated, and we report the $A_i \in \mathcal A_i$ such that $Z_i \in A_i$ (equivalently, we report $I(Z_i \in A)$ for all $A \in \mathcal A_i$). Uninformative censoring of $Z_i$, for example, then means that $\mathcal A_i$ is independent of $Z_i$.

This is a little heuristic and sloppy. We should probably also require that the distribution of $[Z_i \mid Z_i \in a_i]$ is non-degenerate to consider $Z_i$ censored. We also might note that, as defined, this is a generalization of missing data where for $Z_i = (X_i, Y_i)$ one might say $Y_i$ is missing if $a_i = \{x\} \times \mathcal Y$ where $\mathcal Y$ is the sample space of $Y$ and say $Z_i$ is missing if $a_i = \mathcal Z$. When one says "$Z_i$ is censored", if they are following my definition, what they usually mean is "$Z_i$ is censored, but is not missing".

Answer 4

It's important to distinguish censored versus truncated as well as missing data.

Censoring applies specifically to the issue of survival analysis and time-to-event outcomes wherein the event at hand is assumed to have occurred at some time past the point at which you stopped observing that individual. An example is men-who-have-sex-with-men (MSM) and the risk of incident HIV in a prospective study who move and cease contact with study coordinators.

Truncation applies to a continuous variable that evaluates to a specific point at which the actual value is known to be either greater than or less than that point. An example is the monitoring of subjects with HIV and the development of full blown AIDS, CD4 cell counts falling below 300 are evaluated to the lower-limit-of-detection 300.

Lastly, missing data are data that have actual values that are not observed in any sense. Censored data are not missing time-to-event data nor are they truncated.

Answer 5

1. Censored: This is a term used to indicate that the period of observation was cut off before the event of interest occurred. So ''censored data'' indicate that the period of a particular event as not or never occurred