# Time-series classification - very poor results

I am working on a time series classification problem where the input is time series voice usage data (in seconds) for the first 21 days of a cell phone account. The corresponding target variable is whether or not that account cancelled in the 35-45 day range. So it is a binary classification problem.

I am getting very poor results from all of the methods that I have tried so far (to varying degrees). First I tried k-NN classification (with various modifications) and got extremely bad results. This lead me to extract features from the time series - i.e. mean, variance, max, min, total zeros days, total trailing zero days, difference between first half average and second half average, etc. and the most predictive features seemed to be total zeros days and total trailing zero days (using several classification algorithms). This performed the best but the performance was still not very good.

My next strategy was to oversample the negative instances in my training set since there were so few of them. This resulted in more correct cancellation prediction but at the expense of more false-positives.

I'm starting to think that perhaps the time series usage data itself is simply not very predictive (though common sense says that it should be). Perhaps there is some latent variable that I am not considering. Looking at the data also shows some strange behaviour. i.e. some examples show very little or decreasing usage (or sometimes none at all) and do not cancel, and some show a ramp up in usage that do cancel. Perhaps this contradictory behaviour does not generate a very clear decision boundary for a classifier.

Another possible source for error is the fact that many training examples are very sparse (i.e. many days with 0 usage). One idea that I have not tried yet is to split the time series into segments and generate some features that way, but I do not have high hopes.

• For starters, I recommend you try variants of the following.1. Aggregate the usage data by hours/days. 2. From your training data, overlay plots of these time-series to see patterns within the two groups. 3. Explore various time-series/regression models that fit your data well. 4. Use the model coefficients as features for your classifier. – ramhiser Aug 8 '13 at 18:12
• I like your feature extraction technique. I will try it. I have become convinced that my data is too sparse to find patterns, but who knows. THanks! – user1893354 Aug 20 '13 at 14:33

I've had pretty good success applying KNN with Dynamic Time warping as the distance metric.

My research (pdf) suggests that this approach is very difficult to beat. The below schematic is from my python implementation of KNN and DTW on github. Or view in IPython Notebook

If you're training data set is very large, I suggest performing a hierarchical clustering of the distance matrix. Then sampling from desired clusters to produce your smaller training data set. The hclust will ensure you have time series that represent a broad range of time series characteristics in your data.

## The two approaches to time series classification

There are two ways on how to deal with temporal structured input for classification tasks:

1. Dedicated Time Series Model: The machine learning algorithm incorporates the time series directly. I count the KNN with DTW model in this category.
2. Feature based approach: Here the time series are mapped to another, possibly lower dimensional, representation. This means that the feature extraction algorithm calculates characteristics such as the average or maximal value of the time series. The features are then passed as a feature matrix to a "normal" machine learning such as a neural network, random forest or support vector machine. This approach has the advantage of a better explainability of the results. Further it enables us to use a well developed theory of supervised machine learning.

I was also successfully deploying KNN with DTW successfully in the past. However, I was nearly always able to beat its accuracy with a model that uses well designed features. Also, KNN with DTW for binary classifications scales with O(n_t · m_{train} · m_{test}) with n_t being the length of the time series, mtrain and mtest being the number of devices in the train and test set, respectively. This means that the calculations take quite long..

Therefore, I would recommend to pursue a feature based approach.

## tsfresh calculates a huge number of features

The python package tsfresh calculates a huge number of such features from a pandas.DataFrame containing the time series. You can find its documentation at http://tsfresh.readthedocs.io.

You can try it to calculate a huge amount of features. Later you can filter the features for their significance and identify promising candidates.

Disclaimer: I am one of the authors of tsfresh.

• Thanks a lot for your wonderful post. I have a really small time-series (e.g., 10 data points per time series). In that case, would you recommend using tsfresh for my dataset? Thank you :) – EmJ Jul 13 '19 at 3:04