A colleague explained their approach to dealing with left censored data in an analysis, and while I don't think it is the best approach, I am not sure if it is insufficient or not.
My colleague has very large datasets, composed of millions of measurements for different materials. A fair number of these measurements are non-zero values below a uniform instrumental threshold (maybe a tenth to a quarter of each dataset for each material), the majority of measurements are above this instrumental threshold. The question of interest is whether materials differ significantly in (a) their central tendency (mean, median) and (b) their spread (variance).
To deal with their censored data, and the need to log-transform the data so to correct for a strong right-ward skew, my colleague has been effectively converting the censored data to interval truncated data: excluding all values at the minimum instrumental value (i.e. all the censored values), and then removing an equal number of samples from the rightward side of the distribution, so that both sides of a dataset's distribution are equally truncated. Then applying Mann-Whitney tests, ANOVAs, etc to this data, which (for a given material) looks somewhat Normal like after this exclusion and the log-transformation, although missing the tails.
I think this is probably sufficient for saying there are differences in the central tendency (but probably less effective than using a rank-based non-parametric method, which would sidestep this whole need to truncate the data-set at both ends). Has anyone seen any work that uses this symmetric truncation as a way to deal with asymmetrically censored data? I haven't found anything, although informally
I'm not really sure about the variance scenario. Could the fact we are removing different amounts of values from each distribution create some artificial differences in spread, even when the datasets are this large? Maybe? Would a better method be using maximum likelihood methods that directly account for the censoring? (Probably.)
