Standard practice for dealing with U flagged chemistry data

I have a large dataset of environmental chemistry data. Many results are U flagged by either the lab or validators. If I want to use these results to find average values over time I see there are several options for incorporating the U flagged results

1. Substitute the limit of detection (LOD)
2. Substitute the LOD/2
3. Substitute the LOD/sqrt(2)
4. Substitute 0
5. Use the median value (provided fewer than half of the values in the data set are U flagged)
6. Use a trimmed mean
7. Some other methods I don't know about

I feel like there are pros and cons to each of these methods (some bias low, high, skew the variability of the dataset). I do not know what standard practice is for dealing with results below detection limits. Which of these methods (or another) is typical when dealing with chemistry data with missing values?

Edit: Just to clarify, these U flagged values are not literally missing because it's not like nothing is known about them. There is some information: they are greater than or equal to 0 but less than the limit of detection.

• There is a tag for interval-censoring which may be worth browsing. Perhaps experts in it may help you better than I and it might be worth adding the tag to your post. – mdewey Nov 29 '16 at 18:19
• @mdewey interesting. I didn't know that term existed. I'm reading more – Brad Nov 29 '16 at 18:23

Entire books have been written about this, especially Dennis Helsel's Nondetects and Data Analysis (Wiley-Interscience, 2005). Helsel has a NADA package for R, too. I will therefore confine this answer to the most important things I would explain to anyone beginning to analyze environmental data.

1. There are more than 20 different definitions of a "detection limit." Make sure you understand the meaning of the limits that have been given you. Usually they should be considered reporting limits: they are values chosen by the laboratory to limit their liability. They are usually not LoDs, MDLs, PQLS, etc. (which have some relationship to the measurement process). So as not to imply anything unintended, I will call the numbers associated with the U-values "RLs."

2. If by "standard practice" you mean "what everybody does (whether they understand the issues or not," then the answer is to use either 0, RL/2, or RL.

3. If you mean "reasonable practice," then please understand that what everybody does has been thoroughly and persistently criticized in the literature for over 30 years. Sometimes you can get away with these simple substitution methods, especially when the results of your analysis wind up not depending on how you impute the values. But in most cases you cannot. The basic problem is that any fixed imputation method, such as RL/2, collapses a quantity that truly is varying into a quantity that does not vary: that can completely screw up all estimates of variation and at that point there's hardly any use in performing any kind of statistical procedure apart from summarizing the data.

4. Helsel advocates applying nonparametric survival methods. Just negate all the values and pretend they behave like survival times. (It's a clever approach and sometimes works, but it does make fairly strong underlying assumptions about the data and in my experience they don't seem to hold.)

5. A class of maximum likelihood-based techniques works pretty well when there's enough data. I have been adapting these to regression models and more recently to time series models with some success. The challenge lies in making inferences about correlations among data that have a large proportion of nondetects. A simple implementation (which does not allow for variable RLs) is available in the censReg package for R.

6. You're probably best off spending your time developing appropriate graphical methods to display these data. In scatterplots, for instance, use distinct symbols for the four possible combinations of data: both quantified, first ND, second ND, both ND. Plot them at the values given by the RLs so you can see the reporting limits. That gives you the best chance of discovering the parts of the data that will be sensitive to how you treat the nondetects.

7. Learn about the nonparametric methods available for computing upper tolerance limits and upper prediction limits. The beauty of these methods is that often you don't need to impute values to the NDs at all. They are described extensively in the US EPA's Statistical Analysis of Groundwater Monitoring Data at RCRA Facilities. Arguably, this enormous document (nearly 1000 pages including its appendices) embodies "standard practice" across the US and--because it's widely emulated in other countries--throughout the world.

8. Finally, you might as well know the US EPA offers some software that deals with NDs. It's called Pro-UCL. Because it is favored by this regulatory agency, its use is rapidly becoming "standard practice" among consultants--especially among those who have no understanding of statistics. It offers a smorgasbord of procedures for any dataset (ranging from good through horribly bad), enabling any user to pick and choose the statistical results they want. (No comment.) Using it will be labor-intensive--it's basically a big spreadsheet. If you really know what you're doing, there's some value in it; and if you have to submit your results to a US federal or state agency, you might be compelled to use it regardless.

• 1. I mean the detection limit as the validators understood it when they were validating the data. For some Data I have an MDL and RL at least and often an LoQ from the lab too. "non detect"/U flag is determined by either the lab or validators based on other information they have (might also be "UJ" flagged, i.e. between MDL and RL). It's not up to me to second guess the data after it's given to me so if it is non-detect that's the truth going forward. 2. I do mean "what does everybody else do" – Brad Nov 29 '16 at 21:31
• 5. You bring up a good point that as time goes on/different analysis methods/labs are used the detection limit can change and is certainly not constant for all samples. 6. eventually the data are going into creating isopleth maps...which is probably not the right kind of "graph". – Brad Nov 29 '16 at 21:31
• A big part of asking the question is that I heard lots of discussion of simple substitution (RL/2, 0, RL...) methods and how bad they are and in general it just sounds bad. You've given me a lot to think about and read. I'm sure there's a Standard Method book somewhere around here. I should go track one down. Thank you so much for the detailed answer! – Brad Nov 29 '16 at 21:34
• I'm not suggesting second-guessing the lab or the validators--but you need to understand what's going on, lest you use an invalid procedure. In any large environmental dataset it's almost always the case that multiple labs and various validators were involved, so one of the things you have to sort out is the potential for inconsistent meanings of the NDs. This can show up in some especially revealing ways. Sometimes the stated limit is far lower than it should have been and sometimes it is far greater--and good statistical graphics can make those errors very clear. – whuber Nov 29 '16 at 22:19
• I'm curious about (4). What are the assumptions that they "behave like survival times" that often are incorrect? Any random variable that has a CDF has a survival function, so I'm not sure how assuming it behaves like a survival time is limiting, unless you're using a parametric survival model or semi-parametric regression model. There may be other assumptions that are problematic, but I'm not sure on what they are based on the answer. – Cliff AB Feb 19 '17 at 17:58

We see this phenomenon in HIV modeling where CD4 and viral load are frequently below limits of detection, even though a participant is a carrier for the disease. The methods you describe are useful approaches that, while biased, are easy to describe. Let me suggest that another approach you might take is simultaneously calculating means among complete cases, e.g. cases where values did not achieve LLD, then presenting alongside that a proportion/count of cases that did achieve that value.

You suggest many forms of single imputation, which is known not to work. Imputation being a missing value is filled a "best guess" of what it might be. The result is that you tend to underestimate standard errors. To solve this, an approach you might take is a parametric modeling approach where you assume a distribution, such as log-normal, for the concentration values and use an EM-algorithm to simultaneously estimate the shape of their distribution and a range of values that the LLD observations achieved. In doing this, you can appropriately account for the error associated with the unknown LLD values and obtain unbiased estimates of the mean and standard deviation. This would best be presented alongside my earlier suggestion as two approaches to the problem.

Bayesian estimation software like Winbugs or R is adept at performing this kind of inference, but I emphasize that the approach I describe is not actually Bayesian. The EM-algorithm is simply a maximum likelihood approach that uses parametric assumptions about the the data to fill in missing values.