1
$\begingroup$

I am studying ARIMA models and find it hard to validate the model in terms of "it's a good, useful model" and "I shouldn't use that model for prediction".

So at first I started with the easiest model, a random walk, where the model uses the mean value of the series.

Let's say I have the series:

data1<-c(1,1,1,1,1,1,1,1,1,1,1,1,1)
data2<-c(1,2,1,1,1,1,1,1,1,2,2,1,1)
data2<-c(10,-5,33,1,96,45,-978,3,15,8,9,32,12)

All three are random walks, but obviously the values of the first series data1 are better to predict than for data2 or data3.

My question is, how can I quantify this kind of reliability in a metric? (Ideally in a single metric from 1.0 for data1 and ~0 for data3). Probably the variance is the metric, but how can I standardize it to tell if the predictions from the models will be "good" (1.0) or "bad" (0.0)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

First of all, you may be misunderstanding what a random walk is. It is a process $x_t$ the value of which at time $t$ is equal to the initial value plus the cumulative sum of shocks at times $1,...t$, i.e.

$$x_t=x_0+\varepsilon_1+...+\varepsilon_t$$

where the $\varepsilon$'s are random shocks. It could also be expressed as the most recent value plus the most recent shock:

$$x_t=x_{t-1}+\varepsilon_t$$

Your variables data1 through data3 do not seem to be realizations of a random walk process.

The forecast of value at time $t+1$ given the value at time $t$ equals to value at time $t$ in a random walk model:

$$\hat{x}_{t+1|t} = \mathbb{E}(x_{t+1}|x_t) = \mathbb{E}(x_t+\varepsilon_{t+1}|x_t) = \mathbb{E}(x_t|x_t)+\mathbb{E}(\varepsilon_{t+1}|x_t) = x_t+0 = x_t$$

This is because the shock $\varepsilon_{t+1}$ is random and the best prediction is just its mean, zero (random walk with drift would have a constant term extra).

Regarding the last question of quantifying forecasting performance, you could take a look at established measures such as root mean squared error (RMSE), mean absolute error (MAE), mean absolute percentage error (MAPE), mean absolute scaled error (MASE) and the like. MASE will perhaps be the closest to your requirement of fitting in between zero and one. It will not necessarily be smaller than one, but the interpretation will be quite clear: MASE>1 will signal that the out-of-sample forecast does poorer than a naive forecast performs in sample.

Improve this answer
$\endgroup$
12
  • 1
    $\begingroup$ I would simply predict the next value as the mean of all previous values in this example. That would be 0.985. It would be ARIMA(0,0,0) with an intercept. $\endgroup$
    – Richard Hardy
    Commented Mar 7, 2015 at 11:27
  • 1
    $\begingroup$ You should start by specifying a loss function (RMSE is quite a typical choice, but it depends). Given a dataset and a model, calculate the particular value of the loss function. If you have a few models, you can compare their performance in terms of this loss function. Of course, try to avoid overfitting and data mining in the bad sense. $\endgroup$
    – Richard Hardy
    Commented Mar 7, 2015 at 11:44
  • 1
    $\begingroup$ We need one more step here. We have been fitting models in sample. Then RMSE can coincide with standard deviation because we are able to "predict" the mean perfectly. But in reality you will be forecasting out of sample and you will not know the mean. One way to account for this is splitting your sample into a training part and a test part. Estimate models on the training part, make out-of-sample predictions targeting the training part and assess model performance on the training part. $\endgroup$
    – Richard Hardy
    Commented Mar 7, 2015 at 12:09
  • 1
    $\begingroup$ The difference between regression and classification is quite stark here. In classification, you can simply calculate the percentage of correct classifications. There is no such thing in regression. So you can use either absolute measures such as RMSE or MAE or relative measures such as MAPE or MASE. Since the mistake in forecasting can be arbitrarily large, it would be difficult to bound an error measure between 0 and 1. $\endgroup$
    – Richard Hardy
    Commented Mar 7, 2015 at 17:04
  • 1
    $\begingroup$ You must have meant "MASE" instad of "MAPE" in the last line? I would not say that MAPE is especially close to the percentage of correct classifications, but it will tell you how much % on average your prediction is off the true/realized value. $\endgroup$
    – Richard Hardy
    Commented Mar 7, 2015 at 17:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.