2
$\begingroup$

I am trying to identify outliers from a simple time series (ts1; see below) and subsequently fit an ARIMA model on the adjusted ts (w/o the outliers). Appologies, if the questions below are beginner's questions.

I have a question concerning the fitted values for the tsoutliers package in R. I thought that the package identifies outliers iteratively, thereby fitting an ARIMA model. I am not not sure now, how to interpret the output of ?tso $fit$fitted.

What I do:

tt_out <- tso(ts, types=c("AO"))
plot(tt_out)

enter image description here

  plot(tt_out$fit$fitted)

enter image description here

So from these two plots, I can tell that one outlier was correctly identified (at t=172), however, the fitted values still incorporate the outlier. So what are they for? If I want to run ARIMA without the outliers do I need to do this: Replace the outlier by the adjusted value (yhat) and then run ARIMA from the package forecast. However, I thought the goal was to automatically identify and to fit the model directly.

Why is the output of tso fit including the outliers? Do I need to use tso to identify the outliers, and then run ARIMA seperately?

ts1 <- as.ts(c(12,108,72,84,72,108,24,72,60,60,84,36,12,0,24,12,60,48,60,12,12,60,24,12,48,0,36,0,36,24,60,48,36,12,12,12,60,24,84,24,36,48,24,72,12,36,36,36,24,60,48,108,24,48,12,156,84,108,84,108,108,132,96,108,96,60,144,132,144,96,216,252,192,168,252,216,324,420,420,468,528,444,468,516,468,516,456,528,420,684,480,420,600,504,312,420,480,384,528,528,504,540,420,612,348,468,444,444,456,480,528,372,444,372,384,384,444,432,504,492,432,348,408,396,444,408,372,456,372,384,324,480,636,432,456,468,588,336,420,456,384,312,432,360,336,516,444,468,396,360,264,240,252,252,312,372,324,348,396,432,432,384,444,288,384,468,348,348,252,180,300,48,516,444,336,504,300,360,492,492,336,384,372,492,492,456,396,372,396,504,420,468,492,348,396,420,516,552,312,540,528,528,552,468,336,492,444,492,312,564,420,396,588,408,444,432,444,396,396,348,528,504,348,528,408,408,540,360,348,360,300,360,312,276,312,240,324,228,276,180,288,192,240,288,264,288,348,240,384,492,288,312,252,240,216,264,300,216,372,240,252,240,432,348,216,228,336,228,204,144,192,240,168,96,312,312,264,180,204,240,156,156,72,132,144,144,156,84))
$\endgroup$

closed as off-topic by javlacalle, mdewey, Peter Flom Sep 13 '17 at 11:35

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – mdewey, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ The fitted values are the original series minus the residuals of the fitted model. You can get the fitted series without the outlier effects as follows: x1 = ts - residuals(tt_out$fit) - tt_out$effects or x2 = tt_out$yadj - residuals(tt_out$fit), all.equal(x1, x2). $\endgroup$ – javlacalle Sep 13 '17 at 10:18
  • $\begingroup$ Thanks! this is what I was looking for. If you add it as an answer, I will accept it. A follow-up question, why would we be interested in the fitted series with the outlier (if we don't do any prediction afterwards)? $\endgroup$ – Puki Luki Sep 13 '17 at 11:58
1
$\begingroup$

There is a change in the variance in the data beginning around period 60. Tsay argued in his paper that if you don't identify variance change then you can't find the outliers. TSO likely suffered from that pitfall and didn't continue to detect more outliers. It's clearly visible in the data that the variance had a change.

enter image description here

The AR 2 was identified first ,for this model, and then the search for outliers after and then the WLS identified and applied due to the change in variance.

The TSO package is catching only one outlier (at period 173) when there are many others being ignored.

An AR 2 with lag of 1 and 2 with these outliers plus drift might be a good model to use.

Y(T) = 299.59
+[X1(T)][(- 317.32 )] :PULSE 172 I~P00172sfd
+[X2(T)][(+ 223.87 )] :PULSE 133 I~P00133sfd
+[X3(T)][(+ 218.32 )] :PULSE 90 I~P00090sfd
+[X4(T)][(- 217.80 )] :PULSE 199 I~P00199sfd
+[X5(T)][(- 201.40 )] :PULSE 95 I~P00095sfd
+[X6(T)][(+ 195.75 )] :PULSE 250 I~P00250sfd
+[X7(T)][(+ 177.69 )] :PULSE 263 I~P00263sfd
+[X8(T)][(+ 168.82 )] :PULSE 173 I~P00173sfd
+[X9(T)][(+ 166.28 )] :PULSE 213 I~P00213sfd
+[X10(T)[(- 167.90 )] :PULSE 274 I~P00274sfd
+[X11(T)[(+ 162.32 )] :PULSE 137 I~P00137sfd
+[X12(T)[(+ 160.08 )] :PULSE 227 I~P00227sfd
+[X13(T)[(- 155.67 )] :PULSE 205 I~P00205sfd
+[X14(T)[(- 152.18 )] :PULSE 209 I~P00209sfd
+[X15(T)[(+ 148.71 )] :PULSE 104 I~P00104sfd
+[X16(T)[(- 149.40 )] :PULSE 170 I~P00170sfd
+[X17(T)[(+ 137.90 )] :PULSE 221 I~P00221sfd
+ [(1- .474B** 1- .482B** 2)]**-1 [A(T)] sfd

Actual and Cleansed

Tsay Variance Test

Residuals

$\endgroup$
  • $\begingroup$ Thank you for the analysis. The reason why not more outliers are detected is the default value of cval and that I am only interested in additive outliers (AO). Setting a lower cval will result in many more outliers identified. However, I am interested if it makes sense to remove outliers with tso() and then use the adjusted values to fit an ARIMA and why the fitted values in tso() include the outliers. $\endgroup$ – Puki Luki Sep 12 '17 at 14:49
  • $\begingroup$ The 17 listed above are all AO. The variance change isn't an "outlier". It's not about "removing" outliers them, but rather adding them in. You add them in by creating a dummy variables (ie 0,0,0,0,0,0,1,0,0,0,0,0 etc.). $\endgroup$ – Tom Reilly Sep 12 '17 at 15:05

Not the answer you're looking for? Browse other questions tagged or ask your own question.