2
$\begingroup$

In the paper Implications of dynamic factor models for VAR analysis the authors propose a a technique for removing outliers in variables used for dyanamic factors analysis:

"The outlier adjustment entailed replacing observations of the transformed series with absolute median deviations larger than 6 times the inter quartile range [IQR] by with the median value of the preceding 5 observations" (Stock and Watson, 2005)

What happens if the first outliers lie in the first four observations?. For example, consider the following variable:

enter image description here

$\textrm{Median Value} = 41.3$

$\text{IQR} = 3.1$

$6\cdot \text{IQR}= 18.6$

Observation 2 is clearly an outlier. My question is: What is the standard procedure for those values? In this particular case, should we use only the preceding value to replace the observation?

Edit: This is an "ex-post" discussion regarding the method described here to treat outliers, i.e. when it has already been decided to remove them. This is not a discussion about the nature of outliers or the drawbacks of replacing them.

$\endgroup$
5
  • $\begingroup$ This is a (poor) variant of smoothing procedures used by John Tukey and described by him in detail in his 1977 book EDA. Tukey gives several methods to deal with smoothing the ends of a series. Little of this will apply to such a Procrustean, inflexible method of "outlier adjustment" (really a form of trimming). $\endgroup$
    – whuber
    Feb 23, 2023 at 15:23
  • $\begingroup$ @whuber thank you for the reference, I will take a look. In your opinion, what could be a better procedure (or at least more detailed) for smoothing data for factor analysis? $\endgroup$
    – Bertrand87
    Feb 23, 2023 at 15:29
  • $\begingroup$ It depends on the type of data and the objective. Tukey's book, although dated (it focuses on procedures one can actually do with pencil and paper and says nothing about using computers), is still an excellent reference for the concepts and for methods to develop such procedures. In particular, it will help you appreciate the value of re-expressing (transforming) the data, of graphical exploration, and of iterating your analysis until an effective solution is reached. $\endgroup$
    – whuber
    Feb 23, 2023 at 15:31
  • 3
    $\begingroup$ Unless the outlier is in fact wrong, replacing an outlier by anything else replaces a correct value by an incorrect one. I doubt that this will help much. (Smoothing is not the same as replacing observations.) $\endgroup$ Feb 25, 2023 at 21:27
  • $\begingroup$ Clicking on the paper title in your question gives me "page not found". $\endgroup$ Feb 25, 2023 at 21:28

1 Answer 1

1
$\begingroup$

One neat way to deal with the lack of predecessors for the first items in the list is to turn the list into a loop where the last items are effectively the predecessors of the first. That might make the procedure that you are looking at work equally well for all positions in the list, but it is not a procedure that should be adopted without careful consideration.

When dealing with outliers you need to think about what the outliers are and what they can tell you. The categories of outliers depend a lot on context, but a few general categories include mistakes (e.g. values recorded with a typo or an out of place decimal); spurious noise (e.g. where a value was recorded that had been affected by the laboratory door being slammed); plain noise; or something else. If the value is a mistake then fix it with reference to some reliable source (e.g. laboratory records) or discard it. If the value has spurious noise then treat it as a mistake. If the value is affected by ordinary noise then keep it, as it contains the informative signal and the noise that your statistical methods need to deal with. In effect, if the outlier is affected by ordinary noise then it is not an outlier in any sense beyond being at or near the edge of the data and it should not usually be trimmed or adjusted.

If you are dealing with lots of outliers in a dataset then it might be that the data are expressed in a form that makes real values appear to be outliers. It is very common in biomedical experimentation that the values are most nearly normal (or at least symmetrically distributed) when log-transformed.

When a dataset is deemed to contain an outlier that requires some adjustment, think about whether replacing it with a median or mean of the other values is a good plan or whether you should simply omit the outlier. Omission prevents the outlying value from influencing the results and also prevents you from obtaining or giving the impression of having more data-derived information than you actually have. Whatever you do with the outlier should be explicitly declared in the results section.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.