4
$\begingroup$

If we observe two censored distributions, where all observations above a cutoff are set at the value of the cutoff, how can we test whether the observable distributions suggest that the two censored distributions come from the same true distribution?

We could imagine this for income data where incomes above a certain level are reported in a form like '\$250k/year and greater.' Or we could imagine data on campaign contributions where people can only donate $\$X$, but some probably would have donated more in the absence of the cap.

For example: 

d1 <- rnorm(n = 1000, sd = 5)

d2 <- rnorm(n = 1000, sd = 5)

d1 <- ifelse(d1>5,5,d1)

d2 <- ifelse(d2>10,10,d2)

par(mfrow=c(1,2))
hist(d1, xlim = c(-20, 20), ylim = c(0,200))
hist(d2, xlim = c(-20, 20), ylim = c(0,200))

Censored distributions

Improve this question
$\endgroup$
  • 1
    $\begingroup$ If it's recorded as "250+", it is censored, rather than truncated. $\endgroup$ – Glen_b Oct 30 '18 at 3:46
  • 1
    $\begingroup$ Truncated would be the case where you somehow didn't even see the ones that were above your threshold. [For example, imagine a fishing net that only catches fish above a certain size; that's left-truncated] $\endgroup$ – Glen_b Oct 30 '18 at 21:25
  • 1
    $\begingroup$ @jbowman I believe that the censoring here is happening at different points, so I don't think that a KS test is appropriate as the samples are, by definition, not from the same distribution. OP is asking about the true distribution from which the censored observations arose. $\endgroup$ – adityar Nov 13 '18 at 10:16
  • 2
    $\begingroup$ We are assuming that you know the censoring points, right? If so, you just take the less censored distribution, censor it to match the more censored distribution, and then use whatever test for equality of distributions you like. $\endgroup$ – Bill Nov 13 '18 at 14:11
  • 1
    $\begingroup$ @Bill, while that is a reasonable way to get an answer, you're losing some information. For example, just before censoring, the distribution is sloping downward in the first image, so it's a reasonable guess that this will continue moving forward, thus adding more certainty that the second image is produced from the same distribution. $\endgroup$ – adityar Nov 13 '18 at 15:10
0
$\begingroup$

Hi Op is your data assumed to be randomly censored? You could try some survival analysis tests (Log-Rank, Wilcoxon, Wilks/Approximate Chi-Square) or even explore the relationship visually with a QQ plot. Depends on if you are interested in exploring the relationship between the two distributions or testing these two distributions against a specific distribution.

You may find this link helpful https://www.rdocumentation.org/packages/EnvStats/versions/2.3.1/topics/distChooseCensored

Since you are using the rnorm command, if you are interested in testing for normality this paper may also be of use https://www.diva-portal.org/smash/get/diva2:816450/FULLTEXT01.pdf

Improve this answer
$\endgroup$
  • $\begingroup$ What do you mean by randomly censored? In the setting I proposed, the observed value (x_o) is a deterministic function of true value (x_t). If x_t > x_max, then x_t == x_max, else x_o == x_t. $\endgroup$ – Dr. Beeblebrox Nov 15 '18 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.