0
$\begingroup$

I've different data sets that have the feature Volume. This feature represents the absolute number of events. Each observation represents a period (a fixed period such as 15 minutes). You can imagine a time series of 15 Minutes-chunks where each chunk has the feature Volume and represents how often an event occurs within these 15 minutes. For example:

1: 32
2: 33
3: 20
4: 30
...
n: x

where 1,...,n are the 15 minutes chunks.

Now lets say I want to train a machine learning model with the feature volume where I assume a sudden change in the number of events will influence the predicted class. To train my model I need much more data than I have in one dataset. Therefore I will combine datasets. However, the base-rate of the feature volume are totally different in each dataset.

For example:

Dataset 1
Chunc | Volume | Class
1 | 32 | 0 
2 | 33 | 0
3 | 20 | 1
4 | 30 | 0


Dataset 2
Chunc | Volume | Class
1 | 160 | 0 
2 | 140 | 0
3 |  90 | 1
4 |  70 | 1

Dataset 2
Chunc | Volume | Class
1 | 61 | 0 
2 | 53 | 0
3 | 10 | 1
4 | 70 | 0

Now I have the question how to engineer the feature that makes me able to merge the datasets, such as:

Dataset 1
Chunc | Volume | Class
1 |  x | 0 
2 |  x | 0
3 |  x | 1
4 |  x | 0
1 |  x | 0 
2 |  x | 0
3 |  x | 1
4 |  x | 1
1 |  x | 0 
2 |  x | 0
3 |  x | 1
4 |  x | 0

Where the individual chunk does not matter! 
Just the information in Volume and Class are important!

Now I hope you have any suggestions which ways could make sense. I thought about:

  1. Normalization/Standardization of the feature Volume individual for each dataset before merging. Which technique would be appropriate?

  2. A kind of change Point score. Unfortunately I'm very unexperienced in the change point topic. Is there an algorithm that can assign a score?

  3. Very simple, calculate the relative change to the previous chunk for example for dataset 1:

Dataset 1
Chunc | Volume | Relative Change | Class
1 | 32 |            | 0 
2 | 33 |   3.125 %  | 0
3 | 20 | -39.39  %  | 1
4 | 30 |  50     %  | 0

Instead of using the last chunk I could use the mean volume of class 0 or the mean of the last x chunks. It's just to get the basic idea.

Do you have any advise how to approach this problem?

Best!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.