# Interesting class-assignment problem with temporally consistent counts

I have time counts that belong to multi-class categories. Data where a number of experiments was done or not. 0 means a negative outcome, 1 a positive outcome and N means the test was not done. Often, I have a year where experimental protocol was changed so that an experiment was either introduced or discontinued.

For example, class 0-1-1-0-1 belongs to a time series 12, 28, 33, 50, 23.

Now, I have missing values in the class labels. E.g. 0-1-N-N-1 with a corresponding time series. This class possibly contains counts from four groups labeled as 0-1-0-0-1, 0-1-0-1-1, 0-1-1-0-1, 0-1-1-1-1.

Here is an example of dataset:

0-1-1-0-1:  0,  0,  3,  4
0-1-0-0-1:  0,  0, 44, 56
0-1-N-N-1: 15, 29,  0,  0
0-1-0-N-1:  5,  7,  4,  2


So, often there is a temporal consistency in the data and it is possible to map the data just by looking at it. In the given example most of the counts in 0-1-N-N-1 should go towards 0-1-0-0-1. The absolute number of counts should not change.

I wonder what is the name of these kind of problems and what models could I use to takle it. Bayes and self-consistent comes to my mind. Maybe class-assignment?

Here is a way of generating a synthetic dataset. To make it more clear. I have X, but I don't have y to train on. So, I am looking for a way to regenerate y without providing examples as in a supervised learning problem.

%pylab inline
import pandas as pd

import numpy as np

from sklearn.datasets import make_classification
pd.options.display.multi_sparse = False

N_t = 20
N_e = 4

import itertools
classes = tuple(list(itertools.product(['0', '1'], repeat=N_e)))
classes

N_c = len(classes)

noise = np.random.randint(0, 10, size=(N_c, N_t))

signal = np.array([linspace(np.random.randint(0,500),
np.random.randint(0,700),
num=N_t).astype(int)
for i in range(N_c)
])

assert noise.shape == signal.shape

noise.shape

signal.shape

data = signal + noise
df = pd.DataFrame(data, index=classes)

shuffled_index = list(range(len(df)))
np.random.shuffle(shuffled_index)
df = df.iloc[shuffled_index]
y = df.copy()

df_index = df.index.to_frame()
df_index.loc[:8,0] = 'N'
df_index.loc[4:12,1] = 'N'
df_index.loc[8:16,2] = 'N'

masked_index = [tuple(i) for i in df_index.values]

X1.index = pd.MultiIndex.from_tuples(X1.index)
X2 = df.loc[:,6:]
X = pd.merge(X1, X2, how='outer', right_index=True, left_index=True)


Any idea is welcome.

• There's a lot of jargon in this that frankly I don't understand. Is what you're asking the following: "I observe a sequence of 5 distinct, binary X-levels and a corresponding sequence of 4 continuous Y values. Some X levels may be missing and I'm interested in predicting their values."? Jan 8 '19 at 21:12
• The N's in the yticks are the missing values, they have incomplete labels. I want to map the values from incomplete labels to rows with complete labels. N-1 could be maped to 0-1 and 1-1, for example.
– user209249
Jan 9 '19 at 19:32

Here is a very simple approach. It calculates the probabilities for complete labels and then distributes values from incomplete labels accordingly. This solution is easy to grasp, though it does not take all information into account.

def impute_labels(df, target_cols=None, count_cols=None, missing_label='N'):
'''
Missing label imputation.
'''
new_rows = []
try:
df, organism = _strip_timetable(df)
except:
pass
if target_cols is None:
target_cols = df.index.names
if count_cols is None:
count_cols = list(df.columns.get_level_values(0))
df = df.reset_index()
for i in range(len(df)):
row = df.iloc[i].copy()
row[target_cols]
cols_with_missing = row[target_cols].str.contains(missing_label)
cols_with_missing = list(cols_with_missing[cols_with_missing.values].index)
N = len(cols_with_missing)
candidates = _get_candidates(row, cols_with_missing)
priors = _get_priors(df, target_cols, count_cols, candidates)
count = row.loc[count_cols]
try:
priors = pd.DataFrame(data=np.outer(priors, count), index=priors.index, columns=count.index)
except:
return priors, count
new_rows.append(priors)
df_new = pd.concat(new_rows)
df_new = df_new.reset_index().groupby(target_cols).sum()
return df_new.round(0).astype(int)

def _get_candidates(row, cols_with_missing):
candidates = []
N = len(cols_with_missing)
for comb in product(['S', 'R'], repeat=N):
for c, v in zip(cols_with_missing, comb):
row.loc[c] = v
candidate = row.copy()
candidates.append(candidate)
return candidates

def _get_priors(df, target_cols, count_cols, candidates):
priors = df.set_index(target_cols)\
.reindex(pd.DataFrame(candidates)[target_cols+count_cols]\
.set_index(target_cols).index)
if len(count_cols) > 1:
priors = priors.sum(axis=1).to_frame()
priors = (priors / priors.sum()).fillna(0)
priors = priors[priors.values != 0]
return priors


Here is a test case.

def test__impute_labels():
df = pd.DataFrame({'AMP': ['R', 'R', 'S', 'N'],
'CZ': ['N', 'S', 'R', 'N'],
'Counts': [1, 10, 90, 1100],
'Counts2': [0, 0, 0, 1000]}).set_index(['AMP', 'CZ'])
expected = pd.DataFrame({'AMP': ['R', 'S'],
'CZ': ['S', 'R'],
'Counts': [121, 1080],
'Counts2': [100, 900]}).set_index(['AMP', 'CZ'])
result = impute_labels(df, ['AMP', 'CZ'], ['Counts', 'Counts2'])
assert result.equals(expected), result