Can data cleaning worsen the results of statistical analysis?

Question

An increase in the number of cases and deaths occurs during epidemics (sudden increase in numbers) due to a virus circulation (like West Nile Virus in USA in 2002) or decreasing resistance of people or contamination of food or water or increase in the number of mosquitoes. These epidemics will present as outliers which can occur every 1 to 5 years. By removing these outliers we are removing evidence of epidemics which form an important part of forecasting and disease understanding.

Is data cleaning necessary while dealing with outliers caused by epidemics?

Is it going to improve the results or worsen the results of statistical analysis?

Answer 1

It actually depends on the purpose of your research. In my opinion, there could be several:

1. You want to understand what are the typical factors that causes cases and deaths and that are not affected by epidemic periods and factors that causes epidemics (so you are interested in typical not force major probabilities) - in this case you obviously need to remove the epidemic periods from the data, as they are by the purpose of research the outliers to what you would like to conclude
2. You may want to include epidemic changes into your models (regime-switching models, for instance, any good links and model suggestions from the community are welcome here), because you want to know the probability of epidemic period to occur (and also how long it will last), to test the stability and to forecast - in this case you do not exclude epidemic periods, but search for more complicated models rather than go for hammer-econometric-tool $OLS$ or something similar
3. Your primarily goal IS to detect epidemic periods and monitor for them in real-time - it's a special field in econometrics a number of my colleagues are working with at Vilnius University (definitely, you would like to have a lot of epidemic observations to deal with)

So if your primarily goal is something like 2, clearing the data will cause wrong conclusions about the future forecasts, i.e. inaccurate forecasting performance. It is also true that the 2nd case does not necessarily provide better forecasts, but you at least could make conclusions about the probabilities of epidemic periods and their length. This IS vitally important for actuarial mathematicians, so may be you are the one?

Answer 2

I personally wouldn't call this "data cleaning". I think of data cleaning more in the sense of data editing - cleaning up inconsistencies in the data set (e.g. a record has reported age of 1000, or a person aged 4 is a single parent, etc.).

The presence of a real effect in your data does not make it "messy" (to the contrary, the presence of real effects would make it rich) - although it can make your mathematical task more involved. I would suggest that the data be "cleaned" in this way if it is the only feasible way to get a prediction. If there is a feasible way which doesn't throw away information, then use that.

It sounds like you may benefit from some sort of cyclical analysis, given that you say this effect comes around periodically (kind of like a "business cycle").

From my point of view, if you are looking at forecasting something, then removing a genuine effect from that source can only make your predictions worse. This is because you have effectively "thrown away" the very information that you wish to predict!

The other point is that it may be difficult to determine how much of a set of deaths were due to the epidemic, and how much was caused by the ordinary fluctuations.

In statistical terminology, the epidemic sounds like that, from your point of view, it is a "nuisance" to what you actually want to analyse. So you aren't particularly interested in it, but you need to somehow account for it in your analysis. One "quick and dirty" way to do this in a regression setting is to include an indicator for the epidemic years/periods as a regressor variable. This will give you an average estimate of the effect of epidemics (and implicitly assumes the affect is the same for each epidemic). However, this approach only works for describing the effect, because in forecasting, your regression variable is unknown (you don't know which periods in the future will be epidemic ones).

Another way to account for the epidemic is to use a mixture model with two components: one model for the epidemic part and one model for the "ordinary" part. The model then proceeds in two steps: 1) classify an period as epidemic or normal, then 2) apply the model to which it was classified.

Answer 3

To give you a general answer to your question, let me parapharse one of my old general managers: the opportunities of research are found in the outliers of the model you are fitting.

The situation is similar to the experiment performed my Robert Millikan in determining the charge of an electron. Decades after winning the Nobel prize for his experiment, his notes were examined and it was found that he threw out a large number of data points because they disagreed with the results he was looking for. Is that bad science?

If you find a few outliers, then maybe they are due to "statistical abberations". However, if you find more than a few outliers, you need to explore your data more closely. If you cannot attribute a cause for the abberations, then you do not understand the process and a statistical model will not solve your problem. The purpose of a model is to summarize a process, the model will not magically summarize a process the experimenter does not understand.

Answer 4

The role of "data cleansing" is to identify when "our laws (model) do not work". Adjusting for Outliers or abnormal data points serve to allow us to get "robust estimates" of the parameters in the current model that we are entertaining. These "outliers" if untreated permit an unwanted distortion in the model parameters as estimation is "driven to explain these data points" that are "not behaving according to our hypothesized model". In other words there is a lot of payback in terms of explained Sum of Squares by focusing on the "baddies". The empirically identified points that require cleansing should be carefully scrutinized in order to potentially develop/suggest cause factors which are not in the current model. The identified Level Shift in STATE1 for the data you presented in the question below is an example of "knowledge waiting to be discovered".

How to assess effect of intervention in one state versus another using annual case fatality rate?

To do science is to search for repeated patterns.

To detect anomalies is to identify values that do not follow repeated patterns. How else would you know that a point violated that model? In fact, the process of growing, understanding, finding, and examining outliers must be iterative. This isn't a new thought.

Sir Frances Bacon, writing in Novum Organum about 400 years ago said: “Errors of Nature, Sports and Monsters correct the understanding in regard to ordinary things, and reveal general forms. For whoever knows the ways of Nature will more easily notice her deviations; and, on the other hand, whoever knows herdeviations will more accurately describe her ways.”

We change our rules by observing when the current rules fail.

If indeed the identified outliers are all pulses and have similar effects (size) then we suggest the following ( quoted from another poster )

"One "quick and dirty" way to do this in a regression setting is to include an indicator for the epidemic years/periods as a regressor variable. This will give you an average estimate of the effect of epidemics (and implicitly assumes the affect is the same for each epidemic). However, this approach only works for describing the effect, because in forecasting, your regression variable is unknown (you don't know which periods in the future will be epidemic ones)."

This if course requires that the individual anomalies(pulse years) have similar effects. If they differ then a portmanteau variable described above would be incorrect.

Answer 5

One of the most commonly used methods for finding epidemics in retrospective data is actually to look for outliers - many flu researchers, for example, primarily focus on the residuals of their fitted models, rather than the models themselves, to see places where the "day in, day out" predictions of the model fail - one of the ways the model can fail is with the appearance of an epidemic.

It's imperative however that you distinguish between hunting down outliers in your results - probably not the greatest idea ever - and what most people refer to as "data cleaning". Here, you are looking for outliers not because they represent a statistical problem, but because they raise data quality issues.

For example, in a data set I have, there is a variable for onset of disease. For one subject, this date is in November of 1929. Do I think this is correct? No. This indicates a data quality problem that needs to be fixed - in this case correcting the date based on other information about the subject. This type of data cleaning will actively improve the quality of your statistical results.