I have time counts that belong to multi-class categories. Data where a number of experiments was done or not. 0 means a negative outcome, 1 a positive outcome and N means the test was not done. Often, I have a year where experimental protocol was changed so that an experiment was either introduced or discontinued.
For example, class
0-1-1-0-1 belongs to a time series
12, 28, 33, 50, 23.
Now, I have missing values in the class labels. E.g.
0-1-N-N-1 with a corresponding time series. This class possibly contains counts from four groups labeled as
Here is an example of dataset:
0-1-1-0-1: 0, 0, 3, 4 0-1-0-0-1: 0, 0, 44, 56 0-1-N-N-1: 15, 29, 0, 0 0-1-0-N-1: 5, 7, 4, 2
So, often there is a temporal consistency in the data and it is possible to map the data just by looking at it. In the given example most of the counts in
0-1-N-N-1 should go towards
0-1-0-0-1. The absolute number of counts should not change.
I wonder what is the name of these kind of problems and what models could I use to takle it. Bayes and self-consistent comes to my mind. Maybe class-assignment?
Here is a way of generating a synthetic dataset. To make it more clear. I have
X, but I don't have
y to train on. So, I am looking for a way to regenerate y without providing examples as in a supervised learning problem.
%pylab inline import pandas as pd import numpy as np from sklearn.datasets import make_classification pd.options.display.multi_sparse = False N_t = 20 N_e = 4 import itertools classes = tuple(list(itertools.product(['0', '1'], repeat=N_e))) classes N_c = len(classes) noise = np.random.randint(0, 10, size=(N_c, N_t)) signal = np.array([linspace(np.random.randint(0,500), np.random.randint(0,700), num=N_t).astype(int) for i in range(N_c) ]) assert noise.shape == signal.shape noise.shape signal.shape data = signal + noise df = pd.DataFrame(data, index=classes) shuffled_index = list(range(len(df))) np.random.shuffle(shuffled_index) df = df.iloc[shuffled_index] y = df.copy() df_index = df.index.to_frame() df_index.loc[:8,0] = 'N' df_index.loc[4:12,1] = 'N' df_index.loc[8:16,2] = 'N' masked_index = [tuple(i) for i in df_index.values] masked_index = pd.MultiIndex.from_tuples(masked_index) df_masked = df.copy() df_masked.index = masked_index df_masked_grouped = df_masked.groupby(df_masked.index).sum() X1 = df_masked_grouped.loc[:,:5] X1.index = pd.MultiIndex.from_tuples(X1.index) X2 = df.loc[:,6:] X = pd.merge(X1, X2, how='outer', right_index=True, left_index=True)
Any idea is welcome.