58

Definitions vary, and the two terms are sometimes used interchangeably. I'll try to explain the most common uses using the following data set: $$ 1\qquad 1.25\qquad 2\qquad 4 \qquad 5$$ Censoring: some observations will be censored, meaning that we only know that they are below (or above) some bound. This can for instance occur if we measure the ...


47

Methods of censored regression can handle data like this. They assume the residuals behave as in ordinary linear regression but have been modified so that (Left censoring): all values smaller than a low threshold, which is independent of the data, (but can vary from one case to the other) have not been quantified; and/or (Right censoring): all values ...


24

I agree that van der Laan has a tendency to invent new names for already existing ideas (e.g. the super-learner), but TMLE is not one of them as far as I know. It is actually a very clever idea, and I have seen nothing from the Machine Learning community which looks similar (although I might just be ignorant). The ideas come from the theory of semiparametric-...


16

Censoring is often described in comparison with truncation. Nice description of the two processes is provided by Gelman et al (2005, p. 235): Truncated data differs from censored data that no count of observations beyond truncation point is available. With censoring the values of observations beyond the truncation point are lost, but their number is ...


15

You can still estimate parameters by using the likelihood directly. Let the observations be $x_1, \dots, x_n$ with the exponential distribution with rate $\lambda>0$ and unknown. The density function is $f(x;\lambda)= \lambda e^{-\lambda x}$, cumulative distribution function $F(x;\lambda)=1-e^{-\lambda x}$ and tail function $G(x;\lambda)=1-F(x;\lambda) =...


14

This kind of model goes by several names, depending on discipline and topic area. Common names for it are Censored Dependent Variables, Truncated Dependent Variables, Limited Dependent Variables, Survival Analysis, Tobit, and Censored Regression. I am probably leaving out several other names. The setup you suggest where $\min\{y_i,a\}$ is observed is ...


14

Censored vs. inflated vs. hurdle Censored, hurdle, and inflated models work by adding a point mass on top of an existing probability density. The difference lies in where the mass is added, and how. For now, just consider adding a point mass at 0, but the concept generalizes easily to other cases. All of them imply a two-step data generating process for ...


14

As the survival tag is used I'll add an answer offering some examples with a survival analysis flavour. Data point By a data point, we just mean some observation, i.e. the outcome of one or more variables. For instance, we might have the following in a data set: person 1 in our study is a male and dies at age 58. We could consider that a data point. But in ...


12

This is an attempt to answer the request you made in the comments. Independence between $T$ and $C$ vs non-informative censoring In the following, I assume random right censoring. Take a sample of i.i.d. survival times $$(y_1, \delta_1), \ldots{}, (y_n, \delta_n),$$ where $y_i = \min(t_i, c_i)$ is the minimum between the survival time and the censoring ...


9

Are the values always between 0 and 1? If so you might consider a beta distribution and beta regression. But make sure to think through the process that leads to your data. You could also do a 0 and 1 inflated model (0 inflated models are common, you would probably need to extend to 1 inflated by your self). The big difference is if those spikes ...


9

Descriptively speaking, I would offer "a data sample is censored if some observations in it take on, or constitute, the sample's extreme values but their true value is outside the observed sample range". But this is deceptively straightforward. So let's first discuss how we can conclude that a data set is censored, which will naturally lead us to ...


8

Consider the following data on an outcome $y$ and a covariate $x$: user y x 1 10 2 2 (-∞,5] 3 3 [4,+∞) 5 4 [8,9] 7 5 . . For user 1, we have the complete data. For everyone else, we have incomplete data. Users 2, 3 and 4 are all censored: the outcome corresponding to known values of the covariate is not observed ...


7

I was asked to re-post this answer here from my comment at http://doingbayesiandataanalysis.blogspot.com/2012/01/complete-example-of-right-censoring-in.html The specifics of this answer relate to the model in that comment, but the concepts apply to the topic here. The core of the JAGS model for censored data is this: isCensored[i] ~ dinterval( y[i] , ...


7

One option is to include a variable that is 1 if symptom severity was not measured and 0 otherwise, then code all the symptom severities that were not measured as 0. The coefficient on the 0/1 variable will represent the average test score for those that did not have the severity measured and the slope for the severity will be computed based on those that ...


7

We can first reduce this to depend only on certain moments of univariate/bivariate truncated normal distributions: note of course that $ \DeclareMathOperator{\E}{\mathbb E} \DeclareMathOperator{\Var}{Var} \DeclareMathOperator{\Cov}{Cov} \newcommand{\N}{\mathcal N} \newcommand{\T}{\tilde} \newcommand{\v}{\mathcal V} $ \begin{gather} \E[Z_+] = \begin{bmatrix} ...


6

You could try to directly estimate the CDF via a binomial rate smoother ? Here is an idealized example for x stemming from a normal distribution: ci = seq(from=-3,to=3,length=500) X = rnorm(500) Y = rep(NA, 500) for (i in 1:500) Y[i] = as.numeric(X[i] < ci[i] ) plot(ci,Y, type="s") library(mgcv) library(boot) fit=gam(Y~s(ci), ...


6

Can you fit a Cox-PH model to left censored data? Yes. Left censoring can be a subset of interval censoring, and you can fit a Cox-PH model with R's icenReg package, using the ic_sp function. However, rather than blindly plug my own software, I will ask if you really want to fit a Cox-PH model to your data. I'm not saying you don't, but just knowing that ...


6

OP asks about values below "limit of detection" in a quantitation (regression) context. Limit of detection, however is a method performance characteristic that refers to qualitative rather than quantitative tasks. This is why my original answer refers to the lower limit of qantitation (quantification) instead. See e.g. Currie, Pure&Appl. Chem., 67, ...


5

I am not sure if this answers your question but please find below some R code which follows Bender et al. (2005). They describe an approach to simulate a Cox PH regression model with given properties like the proportion of censored events (see line "dat <- data.frame(T = T, X, event = rbinom(n, 1, 0.30))", i.e. 70% of all events are censored). Reference ...


5

In the case of your first equation: $$L(\theta| \mathbf{x} )= [1-F(a-\theta)]^{n_2} \prod_{i=1}^{n_1} f(x_i -\theta) $$ it's assumed you only observe the $n_1$ values of $\mathbf{x}$, but that you also know there were $n_2$ observations censored at $a$. It might have been better for the authors to have written the left hand side as $L(\theta | \mathbf{x}, ...


5

Any method of imputation including multiple imputation is a shot in the dark if you can't take acoount of how the data above 50 are distributed. Since you have 200 variables are any of them correlated with the biomarker? If you could fit a regression for the biomarker as a function of the covariates you could use that model to predict the values for the ...


5

I would follow Henry's tip and check Lyapunov with $\delta=1$. The fact that the distributions are mixed should not be a problem, as long as the $a_i$'s and $b_i$'s behave properly. Simulation of the particular case in which $a_i=0$, $b_i=1$, $k_i=2/3$ for each $i\geq 1$ shows that normality is ok. xbar <- replicate(10^4, mean(pmin(runif(10^4), 2/3))) ...


5

In this case you are ignoring that a payment between £0.01 and £100,000 is not the only choice but that not buying, i.e. £0 is a choice in itself as well. These are two separate selection mechanisms of which the first is continuous (the value of the payment) and the latter is discrete (the yes/no decision of buying at all). You can represent this by assuming ...


5

You could also try a range of plausible values for the unknown cases. Sort of a worst and best case scenario. So try 0, 1, 2, and 3, and see if any of those substantially alter the mean. If they do, you can discuss what the mean is under the best and worst case scenarios. If they do not, you can include that in your discussion as a justification.


5

If people are at risk, then you probably want to count them. If they get censored early, it will not affect things much, but if someone died after 1 week you would surely want to count that (doing anything else could induce survivorship bias). One question could be whether such patients were at risk for some of the outcomes (e.g. if an outcome can only be ...


5

The best way to implement a formula is first to analyze it, because often the way in which it is expressed theoretically is not a good way to compute it. To that end, let's split the sum into two parts: $$\lambda - f(\lambda,C) = \sum_{t=C}^\infty (t-C) \frac{\exp(-\lambda) \lambda^t}{t!} = e^{-\lambda}\sum_{t=C}^\infty t \frac{\lambda^t}{t!} - e^{-\...


5

According to Wikipedia, IQ test results are scaled so that the mean corresponds to a score of 100 points and the standard deviation corresponds to 15 points. Assuming a normal distribution, the proportion of scores that are more than 3 standard deviations below the mean (less than 55) is $\Phi(-3) \approx 0.001349898$. The proportion of scores that are ...


4

The Kaplan-Meier estimator is not biased when a large proportion of individuals are censored. One of the problems we often observe is that the majority of power for the log-rank test is derived from early failure times which are difficult to observe in KM curves. It does mean that the median survival time is an unreliable point estimate. However, the hazard ...


4

The first set of definitions seem right to me. Your adviser's definition two seems to be a conflation of independent and non-informative censoring assumptions. I haven't seen non-informative censoring defined before with reference to the covariate profile. The following text is from "Survival analysis: A self-learning text" by Kleinbaum and Klein (3rd ...


4

In concordance with Greg Snow's advice I've heard beta models are useful in such situations as well (see a Smithson & verkuilen, 2006, A Better Lemon Squeezer), as well as quantile regression (Bottai et al., 2010), but these seem like so pronounced floor and ceiling effects they may be inappropriate (especially the beta regression). Another alternative ...


Only top voted, non community-wiki answers of a minimum length are eligible