0
$\begingroup$

I have some specific concerns regarding data handling and aknowledge the right Time Series forecasting path. I've followed the ARIMA modelling in R guide from the Forecasting: principles and practice, which I think it's one of the best out there. Unfortunatelly most of the guides or tutorials where made with "prepared or adjusted data" so you allways get a nice outcome. For this post I've prepared a public pastebin with the DAILY data set from 2015-01-01 to 2016-09-31 I'm using. I've read a lot of posts but still seems confunsing to me.

1- Since I have skewed data, (ranging from 160.000 to 2.363.188.500, amount of daily sales in my country's currency) do I need to scale it somehow ? My goal is to forecast the next 30 days sales amounts, so is better to leave the data as it is ?

# READ FROM URL
library(data.table)
salesData <- fread('http://pastebin.com/raw/0ebKmSw5',
                header = TRUE,
                data.table = FALSE)
hist(salesData[,2]) # POSITIVE SKEWED TO THE RIGHT

2- I know that working with daily data has some issues for what I'd read. In Rob J Hyndman's blog here he says that:

"Unless the time series is very long, the simplest approach is to simply set the frequency attribute to 7"

I only have 612 data points, so it is safe to set the frequency to 7 ?

amounts <- salesData[,2]
startD <- as.Date("2015-01-01")
tsSales <- ts(amounts, frequency = 7, start = startD)

From here I've used this test to ensure that no transformation is needed, am I correct ?

BoxCox.lambda(tsSales) # = 0.7440572 CLOSE TO 1 -> NO TRANSF. NEEDED
tsdisplay(tsMaxi,main = "ORIGINAL DATA")

3- In this next step, do I really need to do the STL decomposition and Seas.Adjustment on my TS ?

stlSales <- stl(tsSales, s.window = "periodic", robust = TRUE)
adjSales <- seasadj(stlSales)
plot(adjSales, main = "STL DEC. & SEASADL")

4- Sticking with the example, here I check if a Seas.Differentiation is needed, the output is that I need to take a First Diff., but the Box.test states that the series is already stationary, how can I interpret this

# p-value = 2.2e-16 < 0.05 THE SERIES IS STATIONARY
Box.test(tsSales, lag = 7, type = "Ljung") 

# OUTPUT OF THIS TEST: TRUE, THERE IS SEASONALITY SO IT NEEDS TO BE DIFFERENTIATED
fit <- tbats(tsSales)
summary(fit)
!is.null(fit$seasonal.periods) # TRUE

# NO. OF SEAS. DIFFERENCES REQUIRED TO MAKE THE TS STATIONARY
ns <- nsdiffs(adjSales) 
if(ns > 0) {
  diffSales <- diff(adjSales,lag=frequency(adjSales),differences=ns)
} else {
  diffSales <- adjSales
}
tsdisplay(diffSales,main = "SEAS. DIFFERENTIATED SERIES")

5- Next, I do an auto.arima() check and this suggests that the series may have multiple seasonalities. Does this make any sense ? I'm choosing auto.arima because I'm having a hard time deciding the Arima() order by looking at the previous PACF plot.

# PACF INDICATES: AR(6)?
# DIFF. ORDER: d = 1
# MA ORDER: MA(1)?
fit <- auto.arima(diffSales) #SUGGESTS: ARIMA(3,0,1)(2,1,0)[7] (SEASONAL ARIMA)
summary(fit)

6- And finally, after a Residual Analysis, the final forecast and plot

fitResiduals <- residuals(fit)
Acf(fitResiduals) # SHOWS 3 CORRELATIONs OUTSIDE THE LIMITS
Box.test(fitResiduals, lag = 7, type = "Ljung") # p-value = 0.07246 > 0.05 ?
forecastedModel <- forecast(fit, level = 0, h = 30)
plot(forecastedModel, main = "FORECASTS")

And other finally questions are:

  • How can I get to plot these forecasted values with the original series ?
  • How can I interpret these weird date X-axis that is plotting ?
  • I'm getting negative results in the forecasting step, why's the reason ?

Hope you can help me somehow or guide me if I'm mistaken or something. I've tried to make the best reproducible example, sadly I cann't yet upload more urls to my plots because of my low reputation. Thanks in advance.

EDIT ( DATA SET UPDATED )

I was missing some date meassures in my original data set, so here is the updated series (sorry I can't post more links, I don't want to loose the former ones in the original question):

http://pastebin.com/raw/QtqijXaK
$\endgroup$

migrated from stackoverflow.com Nov 7 '16 at 16:21

This question came from our site for professional and enthusiast programmers.

  • 1
    $\begingroup$ Multipart questions are disparaged on SO and this seems to be in serious need of statistical advice rather than a straightforward programming question anyway. I see I'm not the first person with this thought. You might want to check CrossValidated.com to see if there has been another questioner with similar Q's. $\endgroup$ – DWin Oct 25 '16 at 5:56
  • $\begingroup$ Thanks anyways for the replay. I'll try to check again other Q's out there. I did actually saw various posts with more than two Q's so I thought it was ok to do so. $\endgroup$ – Emiliano Oct 25 '16 at 13:10
1
$\begingroup$

The reason you're getting negative results in your forecast is because you aren't forecasting your original series. Instead you're forecasting the differenced series after adjusting for seasonality. See this question for a more in depth explanation. You may want to consider allowing auto.arima() to difference the series automatically, as it will make plotting, extracting fitted values, etc. much easier:

fit <- auto.arima(tsSales)
fc  <- forecast(fit, h = 30, level = 0)
plot(fc)

enter image description here

As for your axes, that has to do with how you've defined your ts object. Since you have daily data with weekly seasonality you may want to consider defining a time series obeject with multiple seasonalities, see msts().

Additional Issues

Your time series is missing 27 days, see the check data.frame created below to see where:

date <- seq(as.Date('2015-01-01'), as.Date(max(salesData$date)), by = "day")
check <- data.frame(date)
salesData$date <- as.Date(salesData$date)
check <- merge(check, salesData, by = 'date', all.x = T)
View(check)

This will be problematic when defining your time series object.

$\endgroup$
  • $\begingroup$ Thanks @Rick, I've done it with auto.arima and that gave me better results. Regarding the axes, if I define a msts like this msts(amounts, seasonal.periods = 7, start = as.Date("2015-01-01")) , it returns a simple ts object because I only have 1 seasonality, it's the only I can identify in my short experience. $\endgroup$ – Emiliano Oct 28 '16 at 16:21
  • $\begingroup$ @Emiliano you also have some issues with missing days in your time series. I'll update my answer to illustrate where. $\endgroup$ – Rick Arko Oct 28 '16 at 17:02
  • $\begingroup$ Yes @Rick you are absolutely right, I've noticed that in later analysis. With the business expert we manage to recover those missing days and I've updated the pastebin url in my original question (down below in EDIT part) if you want to take a look. $\endgroup$ – Emiliano Oct 29 '16 at 13:39
0
$\begingroup$

Emiliano, Take a look at this response. With daily data where there is multiple seasonality it is wise to use dummy variables and regression to model this well. You could consider monthly dummies (11), day of the week dummies(6) and they may not all be significant. You might have the need for trend variables, level shifts, changes in day of the week patterns, outliers, holidays(before and after as well), day of the month effects, long weekends, etc

$\endgroup$
  • $\begingroup$ Just as I said, working with daily data can be problematic. Thanks Tom. Actually my data its meassure by minute (there is a sale every minute) and I summarize it to make it daily. In your opinon, it is better to analyze this daily sales or leave it by minute as it is ? Or maybe make it monthly to get less granularity. $\endgroup$ – Emiliano Oct 26 '16 at 14:42
  • $\begingroup$ Another thing is that currently I can't work with any commercial tool as suggested by that response you pointed, however was very helpful. Is there any R-like implementation of something similar ? $\endgroup$ – Emiliano Oct 26 '16 at 14:48
  • $\begingroup$ It all depends on when you need to take action. If you need to make a decision at the minute level then you should model it at the minute. If not then, yes the daily makes sense. You could set this up in R, but you would need a way to detect when the day of the week pattern changes, find outliers, etc etc. This is why you won't see anything in R on this as it is very complicated. $\endgroup$ – Tom Reilly Oct 26 '16 at 17:59
  • $\begingroup$ You are right that is complicated. I don't know if you had looked at already but I'm kindly asking you, if posible, just to take a look at my data and if you can give me some tips about it. I see that is a strong weekly seasonality, am I right ?, and I'll consider how to proceed from there. $\endgroup$ – Emiliano Oct 26 '16 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.