I have a question on detecting the outliers in a time series like PPI, CPI, inflation,...etc.)
Which method should I use? How can I precisely detect these outliers in a test or a method?
Please mention about all methods.
$\begingroup$
$\endgroup$
3
-
7$\begingroup$ You've added a tag for outliers which links to 1.4k questions. Also, there are entire books on outliers. So, asking for all methods is -- to be frank -- unrealistic. $\endgroup$– Nick CoxCommented Jun 16 at 16:54
-
1$\begingroup$ They're probably not 'outliers' though, the things you mentioned all have fat-tailed distributions, where you can expect values way outside the what looks typical (or rather what you would expect from a gaussian distribution). If you're trying to exclude these points from some analysis you're probably doing the analysis wrong. $\endgroup$– crobarCommented Jun 18 at 7:52
-
$\begingroup$ @crobar thank you for your comments! Please take a look at my another post: stats.stackexchange.com/questions/649370/… $\endgroup$– 1190Commented Jun 18 at 11:27
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
5
An outlier is a surprising point. What points would surprise you?
Make up a rule and apply it.
What rule you make up depends on why you are detecting outliers in the first place. Many times, when people say they want to detect outliers, they don't really need to. Sometimes, they want to discard data. That's a mistake, unless there is data entry error, and data entry error can be detected by eye.
-
3$\begingroup$ (+1) Plus I will add this because Peter possibly won't: In very many (but not all) situations in which a positive outcome seems to show outliers, analysis on logarithmic scale might suggest a more temperate view. $\endgroup$– Nick CoxCommented Jun 16 at 16:57
-
$\begingroup$ I will do nonlinearity tests. And the presence of outliers affect the AR order and the presence of nonlinearity. Thus, I need to detect outliers. $\endgroup$– 1190Commented Jun 16 at 17:30
-
8$\begingroup$ No, that doesn't mean you need to detect outliers (except, again, for data entry errors). You certainly don't want to eliminate outliers because they affect your model. Don't make your data fit the model. Find a model that fits your data, or else admit your data re too messy to be modeled well. $\endgroup$ Commented Jun 16 at 18:31
-
3$\begingroup$ Among many other details, eliminating outliers leaves gaps in time series which you then need (your software) to handle. $\endgroup$– Nick CoxCommented Jun 16 at 20:47
-
1$\begingroup$ Firstly, assume the 'anomalous' data points are factual and investigate whether there is any real world explanation for the event. It may be a compiling sources error or some disruption - an asteroid impinging the nation's capital or what not. Once those are eliminated then this answer comes into play and then the statistical, time series and econometric methods may be brought forward to the data series. $\endgroup$– civitasCommented Jun 17 at 11:08
$\begingroup$
$\endgroup$
6
One definition of outlier is the following:
lower outliers: all points which are less than $Q1 - 1.5 \times IQR$,
upper outliers: all points which are greater than $Q3 + 1.5 \times IQR$
($Q1$ and $Q3$ are first and third quartiles of the distribution, and $IQR$ is the corresponding interquartile range.)
Extreme outliers are defined as:
lower extreme outliers: all points which are less than $Q1 - 3\times IQR$,
upper extreme outliers: all points which are greater than $Q3 + 3 \times IQR$.
More here, in Tukey's fences subsection.
-
1$\begingroup$ For Tukey, these served as rules of thumb for determining which data points should be shown individually on a box plot, or with emphasis. The context was, ~1970, exploratory plotting without computational aids (only an analyst's brain, pen or pencil, and paper). Taking these rules of thumb as criteria for outliers now is somewhere between simplistic and absurd. For one, with time series, it's vital to consider the context of neighbouring values, not just the marginal distribution. For all, including time series, what generating process or processes are plausible is a more important question. $\endgroup$– Nick CoxCommented Jun 17 at 13:48
-
$\begingroup$ Read the book Modern Mathematical Statistics with Applications, by Devore et al. (3rd edition). You will find the above mentioned definition on page 37. This is a textbook knowledge. $\endgroup$– SaneCommented Jun 17 at 15:41
-
1$\begingroup$ Sure, it’s in textbooks and I can cite back at you Tukey’s preliminary editions and published edition of “Exploratory Data Analysis” from 1970, 1971 and 1977. This is still a poor answer to the question IMO. $\endgroup$– Nick CoxCommented Jun 17 at 15:53
-
1$\begingroup$ It is a definition of outlier, although with a very specific purpose in mind. It isn’t at all standard throughout the literature, for many good reasons. $\endgroup$– Nick CoxCommented Jun 17 at 17:54
-
3$\begingroup$ The idea that there is a standard definition for an outlier is itself absurd. Generally, people refer to outliers as out of distribution events, but how could we make a generic statement about them if we can't really make these kinds of generic statements about distributions. $\endgroup$ Commented Jun 17 at 18:24