How to deal with 'cut-off' selection bias/sampling bias? (truncated distribution)

In short

When measuring an outcome with a normal distribution, but whos mean is below the detection threshold, can you still make statements about differences between populations?

Example

Say I only pick up outcomes that are above a certain value. In one population it happens to be the 90th percentile for instance, so I only measure 10% of the outcomes. So In the image below I only pick up outcomes to the right of the vertical bar. Then in another population I do the same measurement and get different outcomes. For both populations I know the mean is below my threshold but for neither I know how far below it is or what the means and variances are.

My question

Can I still make a sensible comparison between these groups in any way? If my samples would be evenly distribubted over the possible values of the outcome I would just have done a t-test.

My thoughts so far

Will the mean I measure still be higher if the actual mean is higher? (I think if the variance is equal then yes, but I might be wrong)

And what if theres not an exact cutoff point but my sampling probability is more like a sigmoidal function with its center at a certain value? The chance I measure an outcome is dependent my measurement so probably there is a 'sampling distribution' which I think should have the shape of the integral of a normal distribution. Maybe when taking this into account I can deduce something more about the true distribution in the population other than just my measurements which are mostly in the tail of the actual distribution.

I have about 20 measured outcomes per population, if I plot its histogram can I deduce something about the actual distribution?

• Is the censoring threshold the same for both groups? Commented Apr 10, 2015 at 19:22
• What exactly do you mean by "pick up" outcomes? Are you saying that the only results available to you are those exceeding the 90th percentile? (This would be a truncated distribution.) Or are you saying that any results that would be less than some threshold are given as "less-than" values? (This would be a censored distribution.) And in either case, what exactly is the mechanism of truncation or censoring? Is it in terms of a fixed value, a potentially variable but known value, a percentile, a random value, or something else? These distinctions are very important!
– whuber
Commented Apr 10, 2015 at 21:13
• @Dimitriy The threshold is the same for both groups.
– Leo
Commented Apr 11, 2015 at 18:05
• @whuber Truncated distribution best describes it.The threshold is unknown. The mechanism is caused by noise in my measurement. I only record a value if 0 is not in the 95% CI. This causes me to have a bias for higher values.
– Leo
Commented Apr 11, 2015 at 18:08

If I had more data, I might be tempted to try interval regression. Here's an example in Stata where I left-censor the log wages below 90th percentile for rural and urban women and compare the means. The first is the truth, and the second is the censored versions:

. use http://www.stata-press.com/data/r14/womenwage, clear
(Wages of women)

. gen ln_wage = ln(wage)

. sum ln_wage , detail

ln_wage
-------------------------------------------------------------
Percentiles      Smallest
1%     1.386294       1.098612
5%      1.94591       1.098612
10%     2.197225       1.386294       Obs                 488
25%     2.397895       1.386294       Sum of Wgt.         488

50%      2.70805                      Mean           2.733084
Largest       Std. Dev.      .5031298
75%     3.044523       4.094345
90%     3.367296       4.127134       Variance       .2531396
95%     3.637586       4.317488       Skewness        .076995
99%     4.007333       4.369448       Kurtosis       3.678885

. gen ln_wage1 = cond(ln_wage<r(p90),.,ln_wage )
(434 missing values generated)

. gen ln_wage2 = cond(ln_wage<r(p90),r(p90),ln_wage )


This means that for observations below 3.367296, we only know that they are in $(- \infty,3.367296]$, which Stata represents as ln_wage1=. and ln_wage2 = 3.367296. Uncensored observations have ln_wage1=ln_wage2:

. list ln_wage* in 420/425, clean noobs

ln_wage   ln_wage1   ln_wage2
3.258096          .   3.367296
3.367296   3.367296   3.367296
3.367296   3.367296   3.367296
3.258096          .   3.367296
3.295837          .   3.367296
3.332205          .   3.367296

. reg ln_wage i.rural

Source |       SS           df       MS      Number of obs   =       488
-------------+----------------------------------   F(1, 486)       =     35.26
Model |  8.34013769         1  8.34013769   Prob > F        =    0.0000
Residual |  114.938825       486   .23649964   R-squared       =    0.0677
-------------+----------------------------------   Adj R-squared   =    0.0657
Total |  123.278963       487  .253139554   Root MSE        =    .48631

------------------------------------------------------------------------------
ln_wage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.rural |  -.2936003   .0494408    -5.94   0.000    -.3907444   -.1964563
_cons |   2.813102   .0258108   108.99   0.000     2.762387    2.863816
------------------------------------------------------------------------------

. intreg ln_wage1 ln_wage2 i.rural, nolog

Interval regression                             Number of obs     =        488
LR chi2(1)        =      10.36
Log likelihood = -147.29026                     Prob > chi2       =     0.0013

------------------------------------------------------------------------------
|      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
1.rural |  -.3468735   .1224948    -2.83   0.005    -.5869588   -.1067882
_cons |   2.755219   .0925769    29.76   0.000     2.573771    2.936666
-------------+----------------------------------------------------------------
/lnsigma |  -.5894893   .1166524    -5.05   0.000    -.8181238   -.3608547
-------------+----------------------------------------------------------------
sigma |   .5546105   .0646967                      .4412588    .6970803
------------------------------------------------------------------------------
434  left-censored observations
54     uncensored observations
0 right-censored observations
0       interval observations


I don't really have a solution to your second questions.

• The mean estimated with interval regression is 2.76 and the actual mean was 2.73, do I read that correctly? That is impressive. Only 55 samples of the 488, thats not a lot. You dont need to know what precentile is known? It look looks like you dont but I dont know how to script Stata.
– Leo
Commented Apr 11, 2015 at 18:26
• @Leo If your data is truncated, then this approach won't work. If it is censored, perhaps a palatable assumption is that the minimum observed value over the two groups is the censoring point? Commented Apr 11, 2015 at 20:35
• @Leo The 2.73 is the overall mean for both rural and urban women in the sample. In the first regression, the constant 2.813102 is the mean among urban women, for rural women the mean is 2.813102 -.2936003 = 2.5195017. You can carry out a similar exercise for the interval regression to get 2.755219 and 2.408345. NB, you are not just using 54 uncensored observations. You are also using the 434 where you only know that log wage is in $(-\infty,3.367296]$, which is the known censoring point. A simpler version of this model is called a tobit and is more widely available in other software. Commented Apr 11, 2015 at 21:08
• You might consider truncated regression, but it probably won't work very well with this degree of truncation. Commented Apr 11, 2015 at 21:33

In the case of censoring, one sensible comparison could be performing quantile regression and comparing both populations at a higher percentile. Say you have 60% of your data below your detection threshold, you could still compare both populations at the 75th percentile. Here's a useful paper about this approach:

@Article{22769433,
AUTHOR = {Eilers, Paul and Roder, Esther and Savelkoul, Huub and van Wijk, Roy},
TITLE = {Quantile regression for the statistical analysis of immunological data with many non-detects},
JOURNAL = {BMC Immunology},
VOLUME = {13},
YEAR = {2012},
NUMBER = {1},
PAGES = {37},
URL = {http://www.biomedcentral.com/1471-2172/13/37},
DOI = {10.1186/1471-2172-13-37},
PubMedID = {22769433}}


According to your last comments, you are dealing with truncation. In that case, if you know the percentile of truncation you still can use quantile regression to compare both populations at a higher percentile. However, you'll need a large sample size for this aproach to work. With a threshold at 80th percentile, 200 samples per group should provide reasonable estimates. Let's see an example:

a<-rlnorm(1000)  #Population A
at<-na.omit(ifelse(a>2.32, a, NA))  #Truncation at 80th percentile, this is your sample
at_imp<-c(rep(min(at), length(at)*4), at)    #We have 20% of values, so we "fill" the other 80% with our minimum

b<-rlnorm(1000, 1)   #Population B
bt<-na.omit(ifelse(b>6.31, b, NA))  #Truncation at 80th percentile, this is your sample
bt_imp<-c(rep(min(bt), length(bt)*4), bt) #We have 20% of values, so we "fill" the other 80% with our minimum

truncated_data<-data.frame(values=c(at_imp, bt_imp), group=c(rep("A", length(at_imp)), rep("B", length(bt_imp))))
original_data<-data.frame(values=c(a, b), group=c(rep("A", 1000), rep("B", 1000)))

#Comparison of results
library(quantreg)
fit1<-rq(values ~ group, truncated_data, tau=0.9)  #Quantile regression for 90th percentile
summary(fit1)  #Results for truncated data

fit2<-rq(values ~ group, original_data, tau=0.9)
summary(fit2)  #Results for original data


If you don't know the percentile of truncation but know the raw value, you could try truncated regression, but with such a high degree of truncation I dont think you would get reasonable estimates.

• Thank you I will have a look at that paper, "many non-detects" does describe my situation. Looking at the quantile regression page on wikipedia its definition is not immediately clear to me. So I would first need to estimate what the cutoff value is, then estimate what percentile I do have for each population and then remove everything below that percentile in both populations and then compare whats left?
– Leo
Commented Apr 11, 2015 at 18:19
• Can you describe with more detail your data? Reading your question it seems like censored data, but then you said in one comment that you think it's truncated. It's not clear to me how you would know the percentiles for which you have data if the threshold is not known. Commented Apr 11, 2015 at 22:06
• I perform a measurement in a population of bloodvessels. Within my measurement area there are potentially hundreds of these vessels, but I only pick up those that have a value that sticks out above the noise. This is not a hard cutoff, but rather an increased probability of picking it up as the value increases. This means that I do no know 'the threshold', but I think I should be able to estimate the threshold or even a sampling probability distribution.
– Leo
Commented Apr 13, 2015 at 10:06
• What are you measuring in that population? Or are you counting individuals? Commented Apr 13, 2015 at 10:16
• I have edited my answer according to the truncated nature of the data Commented Apr 13, 2015 at 12:57