I am working with time-series data and I am trying to classify the Fault happening in the system. The problem is no matter what I try so far, I get 99.79 validation accuracy on the very first epoch. It changes to 99.90 after couple of training runs but nevertheless, it's too good to be true. I have tried the KFold approach and custom metrics (F1) but result is still the same. I also use SMOTE to balance out the dataset, as one of the classes (0) is too prevailing compared to other ones (1 and 2).

My main function:

output_nfs = data_normalization(nf_data.iloc[:, 1:28], 'PCA') #Normalization via StandardScaler
output_afs = data_normalization(af_data.iloc[:, 1:28], 'PCA') 
output_nfs ['Fault'] = nf_data['Fault'].values
output_afs ['Fault'] = af_data['Fault'].values

output_nfs_rec_array = output_nfs.to_records (index = False)
output_afs_rec_array = output_afs.to_records (index = False)

final_data = np.concatenate ([output_nfs_rec_array, output_afs_rec_array])
y=final_data ['Fault'] #target
X=rf.drop_fields(final_data, ['Fault'], False).view (np.float64).reshape(len(final_data), len(final_data.dtype)-1) #Actual datapoints
y_new = [] #Combine all the faults into 3 separate categories: no faults (0), electrical (1), mechanical (2)
for i in range(len(y)):
    if y[i]==0:
    elif (y[i] == 188)|(y[i] ==176)|(y[i] == 315)|(y[i] == 485)|(y[i] == 286)|(y[i] ==707)|\
         (y[i] == 959)|(y[i] ==958)|(y[i] ==817)|(y[i] == 187)|(y[i] == 489)|(y[i] == 632)|\
         (y[i] == 102)|(y[i] ==648)|(y[i] ==687)|(y[i] == 935)|(y[i] == 332)|(y[i] == 846)|\
         (y[i] == 944)|(y[i] == 254)|(y[i] == 181)|(y[i] == 317):
        y_new.append(1) #electrical
    elif (y[i]==604)|(y[i]==603)|(y[i]==958)|(y[i]==154)|(y[i]==162)|\
        y_new.append(2) #mechanical
y = y_new  

from sklearn.model_selection import KFold
n_folds = 10
kfold = KFold(n_folds, True, 1)
scores, members = list(), list()

import keras_metrics as km
from keras.layers import Dropout
labels = ['no faults', 'electrical fault', 'mechanical fault']
def evaluate_model(X_train, y_train, X_test, y_test):
    early_stopping_monitor = EarlyStopping(monitor='val_loss', patience=3)
    model = Sequential()
    n_cols = X_train.shape[1]
    model.add (Dense (35, activation = 'relu', input_shape = (n_cols,))) 
    model.add (Dense (10, activation = 'relu')) 
    model.add (Dense (3, activation = 'softmax'))
    model.compile (loss = 'categorical_crossentropy', optimizer = 'adam', metrics = ['acc'])
    model.fit (X_train, trainy_enc, batch_size=10, shuffle=True, epochs= 100, 
               validation_data=(X_test,testy_enc), callbacks = [early_stopping_monitor], verbose = 1)
    # evaluate the model
    _, test_acc = model.evaluate(X_test, testy_enc, verbose=1)

    return model, test_acc

for train_iX, test_iX in kfold.split(X):
    X_train, y_train = X[train_iX], y[train_iX]
    X_test, y_test = X[test_iX], y[test_iX]
    #Class imbalance is too severe. No Fault prevails. Using smote to balance out electrical and mechanical faults
    from imblearn.over_sampling import SMOTE
    sm = SMOTE (random_state = 12)
    X_train,y_train = sm.fit_resample(X_train,y_train)
    X_train, y_train = utils.shuffle(X_train, y_train, random_state=42)
    X_train, X_test = feature_selection (X_train, X_test, None, 'PCA', None)
    model, test_acc = evaluate_model(X_train, y_train, X_test, y_test)

My PCA function is the following:

pca = PCA (0.95)
X_train_pca = pca.transform(X_train)     
X_test_pca = pca.transform(X_test)
return X_train_pca, X_test_pca

Any suggestions will be appreciated.

  • $\begingroup$ You don't think that the model is overfitting? It just seems very odd that I get almost one at the very first epoch. Several posts have indicated that this is an overfitting issue. $\endgroup$ – Emin Mammadov Mar 21 '19 at 21:11
  • $\begingroup$ I see your point. I honestly doubt that I should be hitting 99% on my validation set immediately. But thank you for the input. Any suggestions for what to look for in data to check for overfitting issue? $\endgroup$ – Emin Mammadov Mar 21 '19 at 21:34

"Too good to be true" is not the definition of overfitting. I am not saying you are not overfitting, but that you do not have to. For example, if you trained a classifier to detect if written text is in English or Chinese, you would very fast get >99% accuracy because this would be a very simple problem. On another hand, there are problems where you would never reach 70%.

As about your problem, you said that you have time series data and you are using $k$-fold cross validation. This means that likely you may be overfitting because using such way of validating the results, you potentially leaking information from the future into your training set. This is not how you validate time-series. Instead you should split your data in such way that you train on the past and test on the future data.

  • $\begingroup$ Thank you. I am using TimeSeriesSplit from sklearn and unfortunately, face different sort of problems right now. But I appreciate your input. $\endgroup$ – Emin Mammadov Mar 23 '19 at 19:42

Not the answer you're looking for? Browse other questions tagged or ask your own question.