Timeline for How to validate random walk model
Current License: CC BY-SA 3.0
14 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 7, 2015 at 17:37 | vote | accept | MikeHuber | ||
| Mar 7, 2015 at 17:31 | comment | added | Richard Hardy | It's up to you to choose what is the most useful accuracy measure. Just be aware of the true meaning of each measure. In classification, there are natural boundaries; you classify some percentage of the cases correctly and the rest incorrectly. In regression, there is no boundary how wrong your forecast can be. You can be off by 1%, 100% or even 1000% (with a bad enough model). | |
| Mar 7, 2015 at 17:29 | comment | added | MikeHuber | Yes I meant MASE in the last line. Sure it's different to classifications, because the classes are nominal and there is no way to calculate a difference to the actual value. But I can use it similar and is probably the closest thing to "how well the model can predict my data in a percentage". | |
| Mar 7, 2015 at 17:22 | comment | added | Richard Hardy | You must have meant "MASE" instad of "MAPE" in the last line? I would not say that MAPE is especially close to the percentage of correct classifications, but it will tell you how much % on average your prediction is off the true/realized value. | |
| Mar 7, 2015 at 17:16 | comment | added | MikeHuber | Yes it's quite different to classification I see. I think MAPE is the closest thing to the accuracy metric in classification tasks? It sounds suitable for my needs because it's independent from a "naive" model and I can use it to compare different time series. So I can use it to tell how "predictable" my data is, which is not possible with MAPE I think. | |
| Mar 7, 2015 at 17:04 | comment | added | Richard Hardy | The difference between regression and classification is quite stark here. In classification, you can simply calculate the percentage of correct classifications. There is no such thing in regression. So you can use either absolute measures such as RMSE or MAE or relative measures such as MAPE or MASE. Since the mistake in forecasting can be arbitrarily large, it would be difficult to bound an error measure between 0 and 1. | |
| Mar 7, 2015 at 16:47 | comment | added | MikeHuber | What still is a bit unclear to me is the interpretation of the MASE, which I tried to use now. An accuracy measure should show me how well my model can predict future data, like an accuracy metric show me for a classification model (in %). But the accuracy measures I found to compare models between different time series like MASE don't provide that information. It always depends on the "naive" model. But sometimes the naive model can already predict future values well and therefore the found model won't have a significant lower error rate but it's still useful to predict future values. | |
| Mar 7, 2015 at 12:09 | comment | added | Richard Hardy | We need one more step here. We have been fitting models in sample. Then RMSE can coincide with standard deviation because we are able to "predict" the mean perfectly. But in reality you will be forecasting out of sample and you will not know the mean. One way to account for this is splitting your sample into a training part and a test part. Estimate models on the training part, make out-of-sample predictions targeting the training part and assess model performance on the training part. | |
| Mar 7, 2015 at 11:53 | comment | added | MikeHuber | so in this example the RMSE would be the standard deviation sqrt((0.985-1)²+(0.985-1)²,...)? | |
| Mar 7, 2015 at 11:44 | comment | added | Richard Hardy | You should start by specifying a loss function (RMSE is quite a typical choice, but it depends). Given a dataset and a model, calculate the particular value of the loss function. If you have a few models, you can compare their performance in terms of this loss function. Of course, try to avoid overfitting and data mining in the bad sense. | |
| Mar 7, 2015 at 11:38 | comment | added | MikeHuber | Alright, I would take the mean value too, because it looks obvious for me in this example. What I wanted to find out is, if there is a metric that quantifies this "obvious" in a single number. So the best thing would be to use the MASE here, by calculating the error for each past value (0.985-1,...), etc. ? | |
| Mar 7, 2015 at 11:27 | comment | added | Richard Hardy | I would simply predict the next value as the mean of all previous values in this example. That would be 0.985. It would be ARIMA(0,0,0) with an intercept. | |
| Mar 7, 2015 at 11:21 | comment | added | MikeHuber | Thanks for clarifying this, but I am a bit confused now what else if not a random walk are those kind of data series. let's say I have the data c(1,1,0.9,1,1,1.1,1,1,1,1,1,0.9), than personally I would not predict the next value with 2, but probably with its mean value from the past. How is that kind of model called and how would it look like in ARIMA(x,x,x). | |
| Mar 7, 2015 at 11:00 | history | answered | Richard Hardy | CC BY-SA 3.0 |