Let's suppose I have a test which gives me the dosage of an analytic in the blood. The results of the assay are in a range of 0 and 1000; all subjects who have a value higher than 1000 will be categorized as ">1000".

The question is: how can I manage this variable since a significant proportion of the cohort will have levels >1000? how can I fit a regression in which the result of the assay is the dependent variable?

  • $\begingroup$ Do I get you right that your dependent variable suffers from measurement error in that it is capped to 1000? Does it make sense to impute the values, i.e. assume a distribution that fits the values below 1000 fairly well and assign random draws from that distribution to the guys above 1000 (given that the random draw is above 1000). $\endgroup$ – E. Sommer Mar 22 at 19:45
  • $\begingroup$ Yes, it is right - the dependent variable is capped to 1000, i.e. those who have values over 1000 will be flagged as ">1000". I don't know if there is any specific methods to manage this type of variable. One option is to categorize them as "1001", but obviously ">1000" may indicates those with 1001 or 20.000, in the same way. Non-parametric regression may be an obvious option, but the question is, is that correct? how can I handle these values? $\endgroup$ – pankevedmo Mar 22 at 19:53
  • $\begingroup$ I suggested to do it the other way round, i.e. impute those unknowns and then run standard regression methods. Your imputed values will be wrong, but this will be less wrong than assuming some fix value. $\endgroup$ – E. Sommer Mar 22 at 19:58
  • 3
    $\begingroup$ Methods of "censored regression," including many techniques of survival analysis, have been successfully applied to this problem, especially in situations like this one where multiple imputation is not a reasonable option. $\endgroup$ – whuber Mar 22 at 20:46
  • $\begingroup$ This is flagged non-parametric, but are you really interested in a non-parametric analysis? Consider if you use a parametric model, you can apply the EM algorithm and obtain estimates and predictions, even beyond the 1000 upper limit of detection. $\endgroup$ – AdamO Mar 22 at 21:38

I'd guess that implicit in your question is not just "what can I do?" but also "what should I do?" The first part is relatively easy to answer, in part because there is so much you can do! The second part will require you to determine what makes the most sense given your setting.

To make sure I understand your question, suppose $Y^*_i$ is the dosage of the analytic in individual $i$, and $X_i$ is some other variable. You observe data $(Y_i,X_i)_i$ for $i=1,\dots,n$, where $Y_i = \min(Y^*_i,1000)$. To make sure we are on the same page regarding terminology, you have a censoring problem. In particular, this is not a truncation problem, which is often confused with censoring (see this question for more on that)

You seem to be interested in a regression model where your dependent variable is censored, as is the case when it's $Y_i$.

1. What you should not do

Before discussing some options, I'd strongly caution against the approach suggested in the first comments of your post, which entails just assigning random values to those with $Y_i = 1000$. To illustrate intuitively why this is problematic, suppose $X_i$ is gender, and suppose all males have $Y_i = 1000$ (i.e. $Y_i^* > 1000$ for all males) and all females have $Y_i < 1000$ (i.e. $Y_i^* < 1000$ for all females). If you just randomly impute $Y_i^* > 1000$ (male data) using $Y_i^* < 1000$ (female data) and run a regression, you will find no effect of gender, when clearly there is one! You may wonder: "Well, what if I impute conditional on $X_i$?" Again, the same problem persists: suppose that males with $Y^*_i < 1000$ have $Y^*_i$ that are like females, but half of males have $Y^*_i > 1000$ whereas all female have $Y^*_i < 1000$. Again, there's a clear difference between males and females, but if you impute all males above $1000$ to have $Y^*_i$ like those males below $1000$, you will erroneously conclude there is no difference between genders, when again you'd expect one!

The lesson here is that a censoring problem should not be solved by imputation, unless you have a very good reason to believe it can.

2. What you may want to do

As you may have guessed based on the above, what you should do will depend heavily on your setup and what you think is reasonable. As a first step, you can always only look at $(Y_i,X_i)$ for $Y_i \leq 1000$. A regression with this data should be fine, except you have to be very clear about the interpretation! Even if the coefficient is the effect of $X_i$ on $Y_i$, here it is that effect for those with $Y^*_i \leq 1000$. Whether that matters depends on your setting. Given that many have $Y^*_i > 1000$, maybe this is not an interesting question.

A general class of regression models such models are called Tobit models, and I'd suggest you check out that page. For more information, check out this note on censored models. These Tobit models typically assume that $$Y^*_i = \beta X_i + \epsilon,$$ where $\epsilon \perp \!\!\! \perp X$ and $\epsilon \sim N(0,1)$. This normality assumption can also be relaxed: see the above note for relaxing that issue using Powell's CLAD Estimator.

Another area you may wish to explore is that of survival analysis, where this issue is very prevalent.

  • $\begingroup$ What is the meaning of $\epsilon \perp X$ please? $\endgroup$ – Good Luck Mar 23 at 0:28
  • 1
    $\begingroup$ Independent -- should have used $\perp \!\!\! \perp $. I edited. Thanks! $\endgroup$ – doubled Mar 23 at 0:35
  • 1
    $\begingroup$ Thank for the suggestions! I'm gonna try out. $\endgroup$ – pankevedmo Mar 24 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.