Trying to capture global time dependences for prediction

Question

I am trying to figure out the best approach for my prediction task. I have a dataset with four variables: year ranging from 2010 to 2022, categorical variables $A$ and $B$, and numeric target variable T. I have numeric data that describes each category in $A$ and $B$, and can be used as embeddings for these instead of the raw categories. Not all categories in $A$ and $B$ occur every year, in fact most combinations occur over only one to two years. The average of my target $T$ seems to show a strong increasing trend. The goal of my problem is to predict target $T$ for future years for a new data sample.

The question is: how can I capture the global trends in $T$ over time while predicting using $A$ and $B$?

Time agnostic models like random forests and boosting would capture the dependencies between $A$,$B$ and $T$ but are not known to capture time trends well. On the other hand, since most $A$x$B$ combinations have data for only one year, I am not sure how I would use time sequence based methods like ARIMA or LSTM.

What approach should I take to my problem? Any help would be greatly appreciated!

PS: My test set may contain unseen categories for A and B, so use of the numeric embeddings is a must.

Answer 1

As it is not possible to work with the data for this problem directly, let me attempt to address your questions in more general terms.

Firstly, you must consider whether your time series T is of a type whereby it can be predicted in its own right using a model such as ARIMA.

As a case in point, let us consider yearly temperature fluctuations. Here is a plot of temperature fluctuations for a region from 1962-2022:

Weather is an example of a time series that has a clear seasonal trend, i.e. it gets warm in summer and cold in winter.

By decomposing this time series, we can see the presence of a seasonality as well as a broader trend of generally increasing temperature, in addition to a random element.

Seasonal

Trend

Random

With a trend and seasonal factor, it is possible to use an ARIMA model to forecast weather patterns for future years.

However, not all time series have such a pattern. In the first instance, I would recommend decomposing your time series T in order to determine if seasonality is present and if a trend exists. Without these attributes, it will be difficult to predict using a time series model such as ARIMA.

With regards to your question about categories A and B – I cannot say whether these factors alone would be of use in predicting the time series T. It could be the case that these variables explain a portion of the variance in your time series, but a significant portion of the variance is down to randomness or other important features that have not been included in the model. Under this circumstance, I would not expect the predictive power of your model to be high.

To summarise, I would firstly recommend decomposing the time series to establish the presence of seasonality and trend. It may be the case that the time series T can be predicted in its own right without having to use A and B as explanatory variables within a model.