Imbalanced data for multiclass classification with ConvNet

Question

I am trying to apply the SMOTE sampling technique to over-sample the minority class of a multiclass (5-class) problem using the convolutional neural network. As far CNN requirement, the input shape for all instances should be of fixed-size. Thus, the input volume for my dataset is prepared to be of shape (1 x 100 x 4) for each instance.

However, smote fit_sample() raised ValueError on passing the input data to resample. Here's my code (and output):

import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense, Conv2D, Flatten, MaxPooling2D
import pickle
from keras.optimizers import Adam
from sklearn.model_selection import train_test_split
import tensorflow as tf
from collections import Counter
from imblearn.over_sampling import SMOTE

tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True))
sess = print(tf.Session(config=tf.ConfigProto(log_device_placement=True)))

#... class distributions
def class_distrib(arr):
    print('-' * 70)
    counter = Counter(arr)
    for c, v in sorted(counter.items() , key=lambda x: x[0]):
        per = v / len(arr) * 100
        print('Class=%s, Count=%d, Percentage=%.3f%%' % (c, v, per))
    print('-' * 70)

filename = 'keras_data.pickle'

with open(filename, mode='rb') as f:
    Input, Label = pickle.load(f, encoding='latin1')  

NoClass = len(list(set(np.ndarray.flatten(Label))))
Instance_Length = len(Input[0, 0, :, 0])

Train_X, Test_X, Train_Y, Test_Y_ori = train_test_split(Input, Label, test_size=0.20)

print('-' * 70)
print('Input  shape before sampling: ' ,Train_X.shape, Train_Y.shape)
print('Class distribution before oversample: ')
class_distrib(Train_Y)

smote = SMOTE('minority')
Train_X, Train_Y = smote.fit_sample(Train_X, Train_Y)
print('Input shape after sampling: ' ,Train_X.shape, Train_Y.shape)

print('Class distribution after over-sampling: ')
class_distrib(Train_Y)

#...to categorical
Train_Y = keras.utils.to_categorical(Train_Y, num_classes=NoClass)
Test_Y = keras.utils.to_categorical(Test_Y_ori, num_classes=NoClass)

print('Label to categorical: ' ,Train_Y.shape)

model = Sequential()
activ = 'relu'
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ, input_shape=(1, Instance_Length, 4)))
model.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ))
model.add(MaxPooling2D(pool_size=(1, 2)))

model.add(Flatten())
model.add(Dense(NoClass, activation='softmax'))

optimizer = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(Train_X, Train_Y, epochs=50, batch_size=64, shuffle=False,
                         validation_data=(Test_X, Test_Y))

Error raised:

----------------------------------------------------------------------
Input  shape before sampling:  (56646, 1, 100, 4) (56646,)
Class distribution before oversample: 
----------------------------------------------------------------------
Class=0, Count=29181, Percentage=51.515%
Class=1, Count=1513, Percentage=2.671%
Class=2, Count=1819, Percentage=3.211%
Class=3, Count=23814, Percentage=42.040%
Class=4, Count=319, Percentage=0.563%
----------------------------------------------------------------------
Traceback (most recent call last):
  File "smote_cnn.py", line 40, in <module>
    Train_X, Train_Y = smote.fit_sample(Train_X, Train_Y)

ValueError: Found array with dim 4. Estimator expected <= 2.

I understand this is due to the input shape (56646, 1, 100, 4), so I tried to flatten the input (n_x x n_y) right before calling smote fit_sample() this way:

#..reshape (flatten) Train_X for SMOTE resanpling
nsamples, k, nx, ny = Train_X.shape
Train_X = Train_X.reshape((nsamples,nx*ny))

This time, CNN's conv2d-layer raised ValueError, as the input shape is changed, output:

----------------------------------------------------------------------
Input  shape before sampling:  (56646, 1, 100, 4) (56646,)
Class distribution before oversample: 
----------------------------------------------------------------------
Class=0, Count=29181, Percentage=51.515%
Class=1, Count=1513, Percentage=2.671%
Class=2, Count=1819, Percentage=3.211%
Class=3, Count=23814, Percentage=42.040%
Class=4, Count=319, Percentage=0.563%
----------------------------------------------------------------------
Input shape after sampling:  (85446, 400) (85446,)
Class distribution after over-sampling: 
----------------------------------------------------------------------
Class=0, Count=29106, Percentage=34.064%
Class=1, Count=1496, Percentage=1.751%
Class=2, Count=1858, Percentage=2.174%
Class=3, Count=23880, Percentage=27.947%
Class=4, Count=29106, Percentage=34.064%
----------------------------------------------------------------------

ValueError: Error when checking input: expected conv2d_1_input to have 4 dimensions, but got array with shape (85446, 400)

Can someone help point out how to use SMOTE here to resample the minority class without changing the input shape to the CNN model? What I am doing wrong?

Answer 1

It's quite clear that after oversampling, you have input of shape $N_{samples}\times N_{features}$ and you didn't reshape it back to 4D tensor.

If your original data is tabular ($N_{samples}\times N_{features}$ plus one column as target) and you want to reshape only once, maybe you can restructure your data processing like this:

1. Train-test splitting first
2. Oversample the minority in training data
3. Reshape and dim-expand the training input to 4D tensor

For the reshaping, you can do some quick test using numpy:

x = np.random.rand(3, 1, 4, 10)
xr1 = x.reshape(3, 40)
xr2 = xr1.reshape(3, 1, 4, 10)

print(np.all(x == xr2))

You would see it prints 'True'. The reshaping works because numpy reads each element by c-like index (can be specified to F or A), and then reshapes the elements to the given shape.