Bad performance of ARIMA model on online buzz data, Any suggestions?

Question

I was wondering if the ARIMA model is constrained to predict online buzz data (time series data).

What I want to do: Use the past round 30 months data to predict next month; and I use Python

Here are my steps for prediction

1. My original data The number of data is 972 (unit = days).

   enter code here

2. I detect the outliers based on z scores and replace it with moving average.

3. I reset the time interval. The data was resampled and aggregated on a weekly basis. The number of data is 137. (unit = weeks)

4. Examining the unit root by the Augmented Dickey-Fuller test (ADF test) and difference if needed

5. Run acf and pacf values,

6. Divide my data into train and test set and the output of ARIMA model

The predicted values are [ 24.23091639] [-14.00353408] [ 6.92498285] [ 6.87718253] The actual values are [[-145. ] [ 143. ] [ 11.42857143] [ 41.90670554]]

It seems to have a poor performance on predicting but I have no idea how to make it more accurate. I have tried several ways including outliers detection, considering the seasonal effects (Too few observations to estimate starting parameters for seasonal ARMA).

I'd like to ask the experts in the community, what would you do if you had this dataset? Of course, it doesn't necessarily have to be limited to traditional time series forecasting models; machine learning methods can also be used

Answer 1

1. You are not saying what you learned from your ACF and PACF plots. Did this lead you to choose a particular ARIMA model? Do consider using an established auto-ARIMA tool: Selecting ARIMA orders by ACF/PACF vs. by information criteria In Python, pmdarima should work reasonably well, since it is a port of the forecast and fable packages in R which IMO constitute the gold standard in automatic time series forecasting.

2. Your time series may well simply have a lot of residual variance, AKA "noise". You can't say that your ARIMA forecasts are "bad" just by comparing a holdout sample to forecasts - if there is a lot of noise, your ARIMA may do the best it can. See How to know that your machine learning problem is hopeless?

3. Finally, you may be surprised at the fact that your ARIMA forecasts vary less than the time series. This is absolutely as it should be. ARIMA, just like any other forecasting/prediction/ML model, tries to disentangle the systematic forecastable components of your data from the unsystematic and unforecastable ones. If there are little systematic dynamics, then your forecast will not vary much. See Canonical duplicate for "Why do predictions vary less than observations?" and links therein.

4. Per above, ARIMA may indeed be the best you can do. Alternatively, consider Exponential Smoothing. We have a collection of forecasting textbooks and resources here: Resources/books for project on forecasting models