Removing outliers from newspaper content analysis

Question

This has been driving me crazy, so I thought I'd come and ask for some help:

I'm doing a study involving content analysis of newspapers. The unit of analysis is each day, adding all stories for the day. Overall, there are 4 variables for each case (i.e. number of stories per day, column-inches total, etc.). After collecting data for a while, I did notice some of the variables in some of the cases were quite different (higher, mostly) than the others, and started considering outliers. Following the basic concept of 3 std. dev., I started checking if some variables did show outliers - and they did.

These outliers are, mostly, days when something unusual happened (a big, breaking story that mattered to that audience of that paper; it's not an error, but an anomaly, I'd say).

So, here's the question:

I do know there's tons of discussions of removing or not outliers from analysis, and don't think there'll ever be an answer for it plain and simple. But, if I DO decide to remove the outliers:

a) Should I remove them before or after describing the frequencies? i.e. "During the X days analyzed, there were 765 stories" - should the 765 include the outliers or not? Should I remove outliers only when comparing means and significant correlations, or remove them altogether?

b) If I do remove the outliers, since they only refer to 1 (2 in very rare cases) of 4 variables I'm collecting for each case, should I remove the whole case altogether or just the outlier variable (leading to different N's in the end for each variable)?

Input on this will be extremely helpful. Thank you SO much in advance!! D.

Answer 1

If the number of stories per day is systematic with the day of the week , or the week of the year , or a particular day in the month , or if there has been shifts in the mean over time or changes in trends over time , or if holidays have a persistant effect then one needs to control for any of these statistically significant factors BEFORE one can think of using some +/- criterion. In addition if there is memory i.e. some form of auto-regressive behavior in the series this must be accounted for in order to establish anomalies. Note that one can't simply look at the acf/pacf of the original series in order to determine seasonality if some of these factors come into play. Pursue Transfer Function Identification integrated with Intervention Detection.

Answer 2

Using three sigma limits is not the way to search for outliers. IrishStat and I have spoken on this numbers of times on CV with respect to time series and in my case the use of influence functions. You can find my AJMMS 1982 paper "The Influence Function and its Application to Data Validation, Gnanadesikan's multivariate analysis book, the work of Ruey Tsay on Time Series and the major papers and book by Doug Martin and his colleagues. Just search CV for the key word "outliers" and "influence functions" and you will ifnd tons of material.

Keep in mind outlier detection should not be used as an automatic data removal procedure. After you find them figure out why they are there.

Answer 3

You've gotten some good answers already, and I don't know time-series analysis well, so let me just make a couple of notes.

1. I'm pretty sure your data aren't / can't be normally distributed (it looks like they're counts) so these may not be outliers--such distributions by nature are often highly skewed.
2. I can't tell if the variables in question are explanatory variables or response variables. Calling something an 'outlier' makes less sense for for an explanatory variable, although such a point can have very high influence over the betas you estimate. This may be a good thing. One way to get more information about how much a given point is driving your results is to fit your model both with and without the point in question and see how that beta changes (a measure called dfbeta).
3. Should these points be outliers, you want to use a different / robust loss function. Note that just throwing out the outliers amounts to a different loss function, just one with less desirable long run mathematical properties. In standard regression contexts, perhaps the most common robust loss function is Tukey's bisquare, but I don't know if it's used with time-series.