# Pre-processing time series data for data mining / predictive modeling input

What are some ways to prepare/pre-process time series data to use the series data as a predictor(s) in a predictive model (classification or regression)? Specifically, what are the methods to be considered in order to

• Get the most predicatively useful signal from the data?
• Reduce the dimensionality of the series

As a concrete example, I have 90 days of ending balance data (amount on deposit in a checking account). I want to use that data to predict if the owner of the account will close it in the next 2 weeks (I have an indicator for this occurring or not 2 weeks after the end of the series).

After reviewing the responses, I think I was looking for a list of commonly used techniques.

• Certainly there are the feature creation that Matt Krause wrote about (each customers balance series are treated separately in all these methods): Things like differences and % changes in the series values each day or week. There are sliding window aggrgations like weekly averages, min, max and standard variations. Also number of increases and decreases, indicators for changes in the balances (absolute or relative of a certain size).
• I have considered fitting a linear or polynomial regression to each serie and using the coefficients in the model as predictors.
• Other things I've wondered about is for each balance series compute the auto correlations for a maximum number of lags and use these values as predictors.
• Cluster the time series into a relatively small number of values and use the indicators for cluster membership as predictors (use dynamic time warping distance and hierarchical clustering for example).

Are there others?

• For example do Fourier Transforms work here? I may post a separate question about them.
• It really depends on your data. It requires domain expertise, and that's not enough. Take a look at the question I asked a while back (search for "Silver"), to see an example of a very hard case. – user765195 Apr 6 '12 at 4:38
• Can you post your data ? perhaps ten accounts along with the indicators ? – IrishStat Apr 6 '12 at 22:41

I agree with what user765195 said: there's no magic bullet here that will work for all your problems. You've got to come up with potentially-useful features based on your domain knowledge. I've never worked in a bank, so take these suggestions with a grain of salt, but how about

• "Volatility" When I've changed banks, I tend to use the new account for a while before I close the old one, since it takes a while for payroll/recurring charges to get moved over. Maybe the variance of the balance (or changes in the variance over shorter time windows) would capture this?

• Transaction Size Taking the derivative of the daily balances would give you an idea of the (net) daily transactions. Maybe people make anamolously large withdrawals before closing out their accounts (e.g., to set up a new account elsewhere).

If I were you, I would start by making a long list of possible features. Carve your data up into a test set, a development set, and a training set. Test out new features on the training+development sets and see what works. Personally, I would throw them all in and see what happens first. There are lots of feature selection algorithms, ranging from the brain-dead but exhaustive (try all possible combinations!) to something like projection pursuit or hill-climbing, which might be more tractable for big data sets.

Then, once you've settled on a model, use the previously-untouched test data to evaluate its performance.

• I think your answer is the best so far. I agree that coming up with features like this makes sense. I was also wondering about other techniques: I will list some in an edit of the original question. – B_Miner Apr 13 '12 at 13:00
• Thanks! From your update, it sounds like you're on the right track! The trends, min/max, etc. all seem like good ideas. I particularly like the idea of doing some kind of frequency analysis. You might want to try that on derviative too; that be a good way to look for the presence of direct deposits/debits, which, based on all the promotions my bank offers, must be somehow correlated with account longevity. The presence/absence of a strong frequency component might be useful by itself, regardless of the actual value. – Matt Krause Apr 13 '12 at 15:05
• Matt, when you say "derivative" are you speaking of % changes period to period (ex. weekly)? – B_Miner Apr 13 '12 at 15:19
• I was actually thinking of a running difference (Day_N - Day_{N-1}). If people have relatively few transactions/day, then that would give you a rough idea of each transaction's size. That said, there's no reason you couldn't do it per week too. – Matt Krause Apr 13 '12 at 15:25
• Gotcha - just wanted to make sure derivative was not something besides a difference or % change. – B_Miner Apr 13 '12 at 15:42

It all depends on how to plan to make predictions after you've finished pre-processing there data. For example, if you expect consecutive observations of a time series $y_t$ to be linearly related to previous measurements:

$$y_{t} = \beta_0 + \beta_1 y_{t-1} + \cdots$$

then you'd better choose a linear transformation. Or, if you expect consecutive measurements to be multiplicatively related to each other:

$$y_{t} = \eta_{0} \cdot y_{t-1}^{\eta_{1}} \cdot \cdot \cdot$$

then a log transformation (and subsequent linear model) would accomplish this since

$$\log(y_{t}) = \log(\eta_0) + \eta_{1} \log(y_{t-1}) + \cdot \cdot \cdot$$

You may investigate various transformations to see which one provides the best predictive power. But, the "best predictor" will depend on your measure of prediction accuracy.

Regarding dimension reduction, your example only seems to involve 2 variables - dimension reduction often refers to the situation where you have a large number of predictors and want to lower the number in a principled way. If you have that situation, you may use standard techniques (e.g. PCA) for dimension reduction, which are not related to the fact that this is a time series.

If you mean dimension reduction in the sense of reducing the number of timepoints, the only types of transformations that will accomplish this will be things like averaging over timepoints (e.g. to calculate weekly averages from daily data). It would be hard to justify doing something like this unless it is drastically easier to fit weekly data than daily data (for example, for rainfall data, it is easier to predict the average rainfall over the course of a week than it is to predict the daily rainfall values).

• I was imagining dimension reduction here in regards to reducing the size of the series. So, instead of 90 variables (one for each day balance), reducing this to a smaller set. Perhaps that could be the first 10 autocorrelation estimates of the series. Or the weekly seasonality indexes. – B_Miner Apr 13 '12 at 12:54