6
$\begingroup$

I have already read

Time Series Forecast: Convert differenced forecast back to before difference level

and

How to "undifference" a time series variable

None of these unfortunately gives any clear answer how to convert forecast done in ARIMA using differenced method(diff()) to reach at stationary series.

code sample.

## read data and start from 1 jan 2014
dat<-read.csv("rev forecast 2014-23 dec 2015.csv")
val.ts <- ts(dat$Actual,start=c(2014,1,1),freq=365)
##Check how we can get stationary series
plot((diff(val.ts)))
plot(diff(diff(val.ts)))
plot(log(val.ts))
plot(log(diff(val.ts)))
plot(sqrt(val.ts))
plot(sqrt(diff(val.ts)))
##I found that double differencing. i.e.diff(diff(val.ts)) gives stationary series.

#I ran below code to get value of 3 parameters for ARIMA from auto.arima
ARIMAfit <- auto.arima(diff(diff(val.ts)), approximation=FALSE,trace=FALSE, xreg=diff(diff(xreg)))
#Finally ran ARIMA
fit <- Arima(diff(diff(val.ts)),order=c(5,0,2),xreg = diff(diff(xreg)))

#plot original to see fit
plot(diff(diff(val.ts)),col="orange")
#plot fitted
lines(fitted(fit),col="blue")

This gives me a perfect fit time series. However, how do i reconvert fitted values into their original metric from the current form it is now in? i mean from double differencing into actual number? For log i know we can do 10^fitted(fit) for square root there is similar solution, however what to do for differencing, that too double differencing?

Any help on this please in R? After days of rigorous exercise, i am stuck at this point.

Edit: Let me paste images from 3 iterations i ran to test if differencing has any impact on model fit of auto.arima function and found that it does. so auto.arima can't handle non stationary series and it requires some effort on part of analyst to convert the series to stationary.

Firstly, auto.arima without any differencing. Orange color is actual value, blue is fitted.

ARIMAfit <- auto.arima(val.ts, approximation=FALSE,trace=FALSE, xreg=xreg)
plot(val.ts,col="orange")
lines(fitted(ARIMAfit),col="blue")

enter image description here

secondly, i tried differencing

ARIMAfit <- auto.arima(diff(val.ts), approximation=FALSE,trace=FALSE, xreg=diff(xreg))
plot(diff(val.ts),col="orange")
lines(fitted(ARIMAfit),col="blue")

enter image description here

thirdly, i did differencing 2 times.

ARIMAfit <- auto.arima(diff(diff(val.ts)), approximation=FALSE,trace=FALSE, 
xreg=diff(diff(xreg)))
plot(diff(diff(val.ts)),col="orange")
lines(fitted(ARIMAfit),col="blue")

enter image description here

A visual inspection can suggest that 3rd graph is more accurate out of all. This i am aware of. The challenge is how to reconvert this fitted value which is in the form of double differenced form into the actual metric!

Edit2: Why it is not so simple. Let me explain by below example.

Actual data with single difference and double difference. enter image description here

Lets go back to actual data by using differences and first value of prior series.

enter image description here

If i use diff(diff(val.ts)) in auto.arima as input data, i get below fitted values. However i do not have first value of first order difference of fitted value and neither i have first data point in fitted value in original metric format! This is where i am struck!

enter image description here

What if i use Richard Hardy's advice and use data from actual series as reference. This gives me negative numbers. Can you imagine negative sales? And to clarify my original numbers do not have ANY negative number and it does not have any returns or cancellation data!

enter image description here

Improve this question
$\endgroup$
7
  • $\begingroup$ What is the problem with the two linked posts? Using the function diffinv is pretty straightforward. Check out this example: x=cumsum(cumsum(rnorm(100))); d2x=diff(x=x,differences=2); y=diffinv(x=d2x,differences=2,xi=x[1:2]) and you will see that x and y are the same. Here y is the undifferenced series that I obtained based on the second differences of x and two initial values of x. $\endgroup$
    – Richard Hardy
    Commented Dec 30, 2015 at 9:14
  • $\begingroup$ Also, you post has several side issues: (1) auto.arima can handle integrated (non-stationary) data and it selects the order of differencing automatically; (2) when comparing which model fits the data best, you have to ensure that the dependent variable is exactly the same in all cases; this does not hold when comparing the model for x versus a model for diff(x); such a comparison is invalid. Also, a visual inspection can always be supplemented by formal measures of forecast accuracy; the picture may be deceptive, so it makes sense to cross validate. $\endgroup$
    – Richard Hardy
    Commented Dec 30, 2015 at 9:18
  • $\begingroup$ Not sure if you read both the linked post in detail. To understand why diffinv() is not useful and cannot be used. please read the description and example given by the guy who asked the question in the second linked question. For your second question, if auto.arima automatically handles non stationary series, then care to elaborate why does fitted vs. actual tend to improve after each subsequent diff()? $\endgroup$
    – StatguyUser
    Commented Dec 30, 2015 at 17:07
  • $\begingroup$ To compare the model for x versus a model for diff(x) i want to reconvert the fitted value into the actual metric. Which is why i posted this question! Also i have added further explanation with step by step screenshot on why above two linked post are of no help. $\endgroup$
    – StatguyUser
    Commented Dec 30, 2015 at 17:08
  • $\begingroup$ Could you explain why the code example in my first comment does not work in your case? The only reason I can see is that you might not have the two initial values; however, you do have the original data, hence, you automatically have its initial values as well. Answering why fit improves with each subsequent diff, I argue that this is not a relevant indicator. Your goal is to fit the original data, not its transformation, hence looking at how well a model fits a data transformation is hardly relevant. $\endgroup$
    – Richard Hardy
    Commented Dec 30, 2015 at 17:16

1 Answer 1

Reset to default
7
$\begingroup$

I found answer in stackoverflow. To summarize instead of doing

ARIMAfit <- auto.arima(diff(diff(val.ts)), approximation=FALSE,trace=FALSE, xreg=diff(diff(xreg)))

we should instead do

ARIMAfit <- auto.arima(val.ts, d=2, approximation=FALSE,trace=FALSE, xreg=xreg)

This d=2 will make sure that forecasted values for future are also in the same metric.

so if i do forecast(ARIMAfit,h=300,xreg=testxreg), i will be able to get future 300 values.

Improve this answer
$\endgroup$
3
  • $\begingroup$ sir i am having temperature data from 1969 to 2013 and i want to predict the temperature for next one year .my file is having this type of format. day temperature 1 35 2 34 . . . . 365 29 1 30 2 34 . . . . 365 29 i required only one order difference.i am facing problem in setting xreg parameter in auto.arima() function.how can i set the xreg parameter please show how you set xreg? and what is the format of the xreg ?(bcz when tried to set it shows me error that it should be vector or matrix) please help me i am stuck here since two weeks $\endgroup$
    – Hiren
    Commented Nov 28, 2016 at 1:14
  • 1
    $\begingroup$ @Hiru, please post this as a separate question along with sample code and data, so that we can have a better understanding of your challenges and respond accordingly. Thanks! $\endgroup$
    – StatguyUser
    Commented Nov 28, 2016 at 8:11
  • $\begingroup$ here i just posted question link thank you and waiting for your response...!! $\endgroup$
    – Hiren
    Commented Nov 29, 2016 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.