1
$\begingroup$

I am very new to R and the forecast package authored by Rob Hyndman. I am working on a time series with 24 samples per hour. I trained a random forest regressor to forecast 6 hour ahead values and am using MAPE(Mean Absolute Percentage Error) on a held out duration as the accuracy metric.

I want to compare its accuracy with standard time series methods like ARMA and ARIMA models.

Time Series Sample

enter image description here

Here is what I have currently. data.csv has 576*16(16 days worth) samples and I wish to measure forecast accuracy on last 3 days.

library(forecast)
pv_data = read.csv("data.csv", header=FALSE)
pv_ts <- ts(pv_data$V2)
train_ts <- window(pv_ts, end=576*13)
test_ts <- window(pv_ts, start=576*13+1)
fit <- auto.arima(train_ts)
accuracy(forecast(fit, h=576*3), test_ts)

This gives me MAPE which is average of h = 1 to 576*3 samples ahead point forecast absolute errors.

Question: How to find the average of h=144 ahead forecast absolute percent error of the estimates of samples in test_ts? Specifically, how to calculate $ \frac {\sum\limits_{T=576*13-h}^{576*16-h} \left\lvert \tfrac{\hat{e}_{T+h}}{y_{T+h}}\right\rvert }{576*3}$ with h=144 where $\hat{e}_{T+h}=\hat{y}_{T+h | T}-y_{T+h}$?

$\endgroup$
  • $\begingroup$ Post the data... $\endgroup$ – Tom Reilly Aug 9 '16 at 13:20
  • $\begingroup$ @TomReilly Created a gist. Please see the update. $\endgroup$ – SPV Aug 9 '16 at 13:28
  • $\begingroup$ What is your question? $\endgroup$ – Richard Hardy Aug 9 '16 at 20:18
  • $\begingroup$ @RichardHardy updated the question. $\endgroup$ – SPV Aug 9 '16 at 20:21
2
$\begingroup$

I'm not really sure what you're trying to achieve here. The forecast at h=144 is just a point estimate whereas the MAPE is a measure of average accuracy over a selection of point forecasts.

As in your case, the sequence of point forcasts from h=1 to h=576*3is used to calculate the MAPE in your current example. If you want to evaluate the accuracy of your 6-hour ahead forecast, why not just look at the forecast error? I.e. $\hat{e}_{T+h}=\hat{y}_{T+h | T}-y_{T+h}$

EDIT: Added a rough outline to a solution of what I think is your problem. You mention calculating the "average" for h=144 but the average is equal to the point estimate because you only have one forecast at this particular horizon.

library(forecast)
pv_data = read.csv("data.csv", header=FALSE)
pv_ts <- ts(pv_data$V2)
train_ts <- window(pv_ts, end=576*13)
test_ts <- window(pv_ts, start=576*13+1)
fit <- auto.arima(train_ts)

#Store point forecasts
fc <- forecast(fit, h=576*3)$mean

# Store actual outcome
y <- window(pv_ts, start=576*13+144, end=576*13+144)

# Store point forecast for h=144
y_hat <- fc[144]

# Calculate ape at h=144
ape <- abs((y_hat-y)/y)
$\endgroup$
  • $\begingroup$ I want to average h=144 ahead forecast absolute percent error of the estimates of samples in test_ts. Specifically calculate $ \frac {\sum\limits_{T=576*13-h}^{576*16-h} APE(\hat{e}_{T+h})}{576*3}$ with h=144. I am not sure how to do that cleanly in R. $\endgroup$ – SPV Aug 9 '16 at 14:37
  • $\begingroup$ I updated the question to make it clear. Please take a look. $\endgroup$ – SPV Aug 9 '16 at 14:47
  • $\begingroup$ So you want to calculate the MAPE from h=1 to h=144 ? Or do you want to calculate $\left|\frac{(\hat{y}_{T+144 | T}-y_{T+144})}{y_{T+144}}\right|$? I'm still not sure what you're trying to achieve. $\endgroup$ – Billywob Aug 10 '16 at 6:45
  • $\begingroup$ I want to calculate MAPE only for h=144 which is the average of $\left|\frac{(\hat{y}_{T+144 | T}-y_{T+144})}{y_{T+144}}\right|$ $\endgroup$ – SPV Aug 10 '16 at 6:55
  • $\begingroup$ Updated answer to what I think you want to do. However, like I mention, calculating the average of $\left|\frac{(\hat{y}_{T+144 | T}-y_{T+144})}{y_{T+144}}\right|$ doesn't make any sense because this is just one number. $\endgroup$ – Billywob Aug 10 '16 at 7:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.