# Time-series classification - very poor results

I am working on a time series classification problem where the input is time series voice usage data (in seconds) for the first 21 days of a cell phone account. The corresponding target variable is whether or not that account cancelled in the 35-45 day range. So it is a binary classification problem.

I am getting very poor results from all of the methods that I have tried so far (to varying degrees). First I tried k-NN classification (with various modifications) and got extremely bad results. This lead me to extract features from the time series - i.e. mean, variance, max, min, total zeros days, total trailing zero days, difference between first half average and second half average, etc. and the most predictive features seemed to be total zeros days and total trailing zero days (using several classification algorithms). This performed the best but the performance was still not very good.

My next strategy was to oversample the negative instances in my training set since there were so few of them. This resulted in more correct cancellation prediction but at the expense of more false-positives.

I'm starting to think that perhaps the time series usage data itself is simply not very predictive (though common sense says that it should be). Perhaps there is some latent variable that I am not considering. Looking at the data also shows some strange behaviour. i.e. some examples show very little or decreasing usage (or sometimes none at all) and do not cancel, and some show a ramp up in usage that do cancel. Perhaps this contradictory behaviour does not generate a very clear decision boundary for a classifier.

Another possible source for error is the fact that many training examples are very sparse (i.e. many days with 0 usage). One idea that I have not tried yet is to split the time series into segments and generate some features that way, but I do not have high hopes.

• For starters, I recommend you try variants of the following.1. Aggregate the usage data by hours/days. 2. From your training data, overlay plots of these time-series to see patterns within the two groups. 3. Explore various time-series/regression models that fit your data well. 4. Use the model coefficients as features for your classifier. Aug 8, 2013 at 18:12
• I like your feature extraction technique. I will try it. I have become convinced that my data is too sparse to find patterns, but who knows. THanks! Aug 20, 2013 at 14:33

I've had pretty good success applying KNN with Dynamic Time warping as the distance metric.

My research (pdf) suggests that this approach is very difficult to beat. The below schematic is from my python implementation of KNN and DTW on github. Or view in IPython Notebook

If you're training data set is very large, I suggest performing a hierarchical clustering of the distance matrix. Then sampling from desired clusters to produce your smaller training data set. The hclust will ensure you have time series that represent a broad range of time series characteristics in your data.

## The two approaches to time series classification

There are two ways on how to deal with temporal structured input for classification tasks:

1. Dedicated Time Series Model: The machine learning algorithm incorporates the time series directly. I count the KNN with DTW model in this category.
2. Feature based approach: Here the time series are mapped to another, possibly lower dimensional, representation. This means that the feature extraction algorithm calculates characteristics such as the average or maximal value of the time series. The features are then passed as a feature matrix to a "normal" machine learning such as a neural network, random forest or support vector machine. This approach has the advantage of a better explainability of the results. Further it enables us to use a well developed theory of supervised machine learning.

I was also successfully deploying KNN with DTW successfully in the past. However, I was nearly always able to beat its accuracy with a model that uses well designed features. Also, KNN with DTW for binary classifications scales with O(n_t · m_{train} · m_{test}) with n_t being the length of the time series, mtrain and mtest being the number of devices in the train and test set, respectively. This means that the calculations take quite long..

Therefore, I would recommend to pursue a feature based approach.

## tsfresh calculates a huge number of features

The python package tsfresh calculates a huge number of such features from a pandas.DataFrame containing the time series. You can find its documentation at http://tsfresh.readthedocs.io.

You can try it to calculate a huge amount of features. Later you can filter the features for their significance and identify promising candidates.

Disclaimer: I am one of the authors of tsfresh.

• Thanks a lot for your wonderful post. I have a really small time-series (e.g., 10 data points per time series). In that case, would you recommend using tsfresh for my dataset? Thank you :)
– EmJ
Jul 13, 2019 at 3:04

In time series problems, the performance often drops rapidly when the predicted phenomenon is distant in time from the train data. I am describing a related issue in this post: why performance drops so fast when test set is distant from train?:

This is generally logical: in the problem described here, we try to predict human behavior (a decision to cancel the mobile telephone account). Such a decision is very possibly influenced by some very recent events (such as an irritating technical problems, or a frustrating helpline call) rather than the events more distant in time. My hypothesis: the main cause for the cancellations in days 35-45 could be the receipt of the first invoice, on day 31.

There may be few day's gap between the causal event (such as frustration upon receiving the invoice) and the target action (cancellation). This short period, characterized by an increased user frustration, could indeed be reflected in change of behavior (between days 31 and day 35), and that could be detected by classifier. But since the train set contains data up to day 21 only, it misses those most vital signals. In other words, the train has no causal relationship with the target variable.

Solution: get hold of the train data directly preceding the target action. Then try to predict the target action 1-2 day in advance. Chances to predict the same 15 days in advance are slim.

For this reason, the literature recommends to retrain the time series models optimally every time before prediction, on the freshest data. In this particular case, it means the we need the call data from the days immediately preceding the cancellation event. Here is an article on this: ML time series forecasting the right way