There were several samples BLQ because of the lower limit of quantitation (LLQ) of the method, e.g. 5 ng/ml or less. Using the statistical program PRISM6 I marked these values together with the outliers (determined with the Rout method, 1% rule, before doing the calculations). The question was raised by the laboratory if excluding BLQ values from the analysis would cause a bias on the data.

  • 2
    $\begingroup$ It depends on the statistical analysis: perhaps you could amplify your question and tell us a little more about what you are trying to do? In most cases--such as estimating averages, fitting distributions, calibrations, etc--dropping extreme values (and by definition, BLQ values are at the lower extreme) introduces substantial bias. Are you familiar with Dennis Helsell's work on nondetects? $\endgroup$ – whuber Apr 2 '14 at 23:42
  • $\begingroup$ Why are you censoring data? Can you just use all of the data and their uncertainties to get the answers you need? Why are are expunging outliers? Do you have subject-specific information that leads you to do this? The overall question you need to answer is do you get a better answer by throwing away information? $\endgroup$ – Thomas Apr 3 '14 at 10:48
  • $\begingroup$ You may be interested in stats.stackexchange.com/questions/30728/… where the questions started after values had been censored. $\endgroup$ – cbeleites unhappy with SX Apr 3 '14 at 11:26
  • $\begingroup$ Also, please give us your context: are you the one who sent analyses to the lab (then congratulations on a good lab that didn't censor the data!), or are you in the lab and consider censoring data before reporting it to the customer? $\endgroup$ – cbeleites unhappy with SX Apr 3 '14 at 11:28
  • $\begingroup$ Oh, and welcome to cross validated! $\endgroup$ – cbeleites unhappy with SX Apr 3 '14 at 11:42

Yes, left censoring data or even truncating data will skew your distribution. It may also bias the data analysis. Whether and to what extent depends on how much of the data is affected and what analyses come afterward.

  • outliers: There are several questions here about outlier detection. Good starting point:

    There is no simple sound way to remove outliers.

    From https://stats.stackexchange.com/a/37876/4598

    I think the most general statements I'd make without knowing more details about your data, samples (cases), models and application are:

    • If you know that a gross error occured, deleting/excluding is fine.
    • If you know the physical/chemical process that produced the outlier, and are sure it is disturbing your data (e.g. cosmic ray spikes in Raman data, bubbles in flow cell), then you may exclude these as well.

    Note that I prefer if the subjectivity of such decisions is well explained over cases where pseudo-objective outlier detection is used that in fact just hides the subjectivity (which is not true for all cases where automated outlier detection is done, sometimes it is important and well justified)

  • left censoring: Let me ask back: would there be any problem with not censoring your data?

    • If you are in the analysis lab and think whether you should report the values to your customer: always report them. You can easily mark below LOQ or even below LOD results as such.
    • If you are analysing the data from the lab: I'd usually include all values. LOD and LOQ are useful telling you when the results become really unreliable (for LOQ i.e. relative error is very large). Nevertheless, these values carry more information than ignoring them (pretending they were never measured) or censoring/replacing by some arbitrary value (see https://stats.stackexchange.com/a/30739/4598 for illustrations).

However, IMHO you can set up criteria that restrict the cases you analyse further (e.g. only values > critical value, no outliers according to automated test). In that case you should usually report

  • how many cases of your training data were excluded, and
  • also for the validation data, how many cases were "rejected" due to violation of which inclusion rule.

Note that this whole business is a non-issue for certain types of exclusion (e.g. quality filters for cosmic ray spikes in Raman spectra, for spectra of the sample holder but not the sample, for spectra with too high or too low intensity according to the instrument type, etc.) and a serious issue for others (e.g. tumor diagnostic studies excluding large numbers of borderline cases, see e.g. C. Beleites, R. Salzer and V. Sergo: Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues, Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22. for a discussion of issues with this approach which will generate artificially easy problems.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.