How to analyse results of classification for time series + sliding windows

Here is my context:

I have a time series composed of only 1 features. I want to be able to classify between two classes. To get more information out of these data, I am using a sliding time windows.

For example, if my time series contains 1000 observations with, I use a sliding window of 20 observations, and I iterate through all 1000 observations. I associate iteration t of my time series with the first value of the window (it t=1: window have t in [1,21]; if t=150, window have t in [150,170]) For each window of 20 samples, I extract additional features (let's say mean and variance for simplicity).

Doing so, I have a new data set of size 1000*2. Then, I train a classifier on this data set. I do not use time as a feature.

In order to evaluate the model, I have a test set (a time series of 200 observations). I do the same process with the time window to get a data set of 200*2. From there, I compare the true class from predicted class and create my confusion matrix.

To summarise quickly the whole process:

• Transform a time series x of N observations and f features into a training set X of N observations and F features with sliding time window
• Train classifier with X
• Transform another time series y into a test set Y (same process as 1) )
• Predict class of Y with trained model
• Create confusion matrix

Now, here is my problem: When I get my confusion matrix, I get a lot of false positive. The reason is that if the class are extremely different in the time series, the statistical value are kind of 'wrong' when class transition occurs. It is kinda hard to explain, so I made a figure of the problem: Here is a time series with two classes (blue and green points), and I superposed the mean obtained by the sliding time window. It is possible to see that a 'trantion phase' happen to the mean when switching classes. The problem is that I got a lot of misclassification because of that. For example, it is going to classify green class at indexes 12,13,14 as blue

My problem is that, the classifier isn't technically 'wrong' , as those examples exists in the sliding window, but it is not visible in the confusion matrix.

My point is: how would it be possible to take into consideration this phenomenon in the confusion matrix ?

I hope I am clear enough, and thanks in advance for any opinion on this !

Ok, so the workaround I am using right now is the following. I don't know if it is optimal, but it is the best I could think of.

Just to clarify, I want to compare classification algorithms. To do so, I use the same data sets for each algorithms, and have the positive rate and negative rate of the data sets. Thus, I would like to respect these rates as much as possible in the confusion matrix.

For now, here is what I do: When I do the confusion matrix for, let say, class 1. I do my predictions based on my sliding window. Here are my possibilities: For an observation t, the sliding window is composed of [t,t+window size] If model predict class 1 and at least class 1 is in time window: True negative If model predict class 1 and no class 1 in time window : false negative If model predict class 2 and at least class 2 in time window: true positive If model predict class 2 and no class 2 in time window: false positive

The difference comes from the fact that we take the most favourable label in the whole window, instead of just the observation t.

The problem I have with that is that it can be seen as a bias for the result, by taking the most favorable case.

What do you think ?

This transition period fo your timeseries can be accounted for in your model creating features that represent well the increasing or declining of the timeseries. For your example figure, a feature $$mean(y[t-w//2 : t] - y[t-w : t-w//2])$$ would represent with the most recent half of the series window is bigger or smaller on average than the first half, that is, if it is in a positive trend or not. Note that while I used the average for this example it could be more adequate to use other functions such as absolute value.

I assume this figure example is a simplification of your real problem and if it is the case that there are more nuances to the problem, additional features will be necessary. But it all goes in the direction of creating good features that represent what you model needs to know. For example, if there are "critical values" that indicate some category, maybe a "max" or "min" feature are necessary.

For the second part, it also seems to me that there is bias in this type of modelling decision.