0
$\begingroup$

I am trying to compare some method of forecasting on wind speed data that I have.

Validation method I saw often use RMSE and MAE. So I am planning to observe these values on different forecast method.

It is just in most paper I read the picture graph they only show curve fitted not out of sample graph. So I am kinda confused , could I really just use observation of fitted graph?

Another question if you do not mind, is there recommendation of other method that I could compare with ARIMA? Currently SVM and Holt-Winters are a candidate but, I want to see if there are other method better suited for the 1 year wind speed data that I have.

Thanks in Advance.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ If you are trying to assess the potential accuracy of your model when predicting future weather, then testing it against out-of-sample test data is a good idea. If you are going to start by model selection and tuning, then (cross-)validation would be another useful tool too $\endgroup$
    – Henry
    Commented Apr 22, 2021 at 1:08
  • $\begingroup$ Is cross validation is the same as curve fitting or in-sample fit or it is a different method from both in-sample and out of sample? $\endgroup$
    – Iyo Widiastomo
    Commented Apr 23, 2021 at 11:30
  • $\begingroup$ Cross validation is not the same as curve fitting: models are fitted to the training data, and then assessed against the separated validation data, which allows choices between models and tuning of hyperparameters (e.g. dropping variables, regularisation or more) $\endgroup$
    – Henry
    Commented Apr 23, 2021 at 11:49

2 Answers 2

Reset to default
1
$\begingroup$

Out of sample is definitely necessary when looking across models because some models will fit much closer to the samples than others. But these same models can perform much worse for your forecasting problem. There are some in-sample measures such as AICC/BIC which are appropriate for certain models but doesn't apply to everything. At the end of the day you want to forecast so you should judge based on the forecast.

In terms of other models to try:

  1. fbprophet is a nice plug and play model
  2. A Mean/Median is a good baseline to have and can be best a lot
  3. Likewise a seasonal naive AKA just repeat your last year of data can be good
  4. LSTMs are ok but less plug and play
  5. I have some old code you could play around with https://github.com/tblume1992/LazyProphet

Time series literature has a ton more and the hyndman book is a good place to start although it is in r: https://otexts.com/fpp2/

Improve this answer
$\endgroup$
0
$\begingroup$

+1 to Tylerr's answer and Henry's comment.

Do not use in-sample fit as a criterion for a good model. In-sample fit can always be improved by making your model more complex, but that has nothing to do with forecast accuracy.

For instance, try simulating some completely IID random noise, then first fit a model that is just the overall mean, then Single Exponential Smoothing, then Exponential Smoothing with seasonality, finally Exponential Smoothing with seasonality and trend. Each successive model will have a better in-sample fit than the previous model, since they are nested. But each model will also give worse forecasts than the previous one, because they are overfitting more and more to dynamics that simply aren't there.

Information criteria try to mitigate this overfitting, but they cannot be compared between model classes.

Wind speed may well have some yearly seasonality. If you have daily data, then these are quite long seasonal periods; ARIMA does not do well on such long seasonalities. You could look at BATS or TBATS models, which do a better job. They are available in the forecast package for R.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.