LSTM - Multiple Time Series, degrading accuracy

Question

I'm trying to make a LSTM model for detecting failures on a physical system, by supplying 27 features of sensor data. I've inputted three disjunct timeseries, each beginning with "normal" operational sensor readings before a failure occurs (each timeseries contains a new type of failure). Each timeseries has 10200 datapoints after splitting the sets in training and testing sets.

My main problem is that the accuracy gets continually degraded and suffers especially during a transition from one timeseries to the next (see attached figure). I'm curious if my implementation has methodical errors, and would be very grateful if someone would point any mistakes.

I've tried changing batch and epoch size, experimenting with different optimizers and layer combinations - but it still shows very strange transitional behaviour during the change from one timeseries to another.

Code:

    def model_initialize(self, datasets):
    num_datapoints = 0 

    for k in datasets:

        X = datasets[k].iloc[:, :-6] # Prediction variables
        y = datasets[k].iloc[:, -6:] # 

        num_datapoints = datasets[k].shape[0] + num_datapoints

        input_dim = 27
        # One timestep at a time, ensure that statefulness is enabled. 
        timesteps = 1
        batch_size_c = 12
        epochs = 100

        split_num = 0.7
        train_elements = 0
        test_elements = 0 


        if(self.variable_split == 0): 
            deletedRows = 0

            split = int(len(datasets[k])*split_num)
            train_elements = split
            test_elements = len(datasets[k])-split
            while True: 
                if ((train_elements%batch_size_c == 0) and (test_elements%batch_size_c == 0)):
                    print("Sets are now divisble by batch size - deleted " + str(deletedRows) +" rows")
                    break
                datasets[k].drop((datasets[k].shape[0]-1),inplace = True)
                split = int(len(datasets[k])*split_num)
                train_elements = split
                test_elements = len(datasets[k])-split
                deletedRows = deletedRows + 1


        X_train, X_test, y_train, y_test = X[:split], X[split:], y[:split], y[split:]


        # Feature Scaling

        scaler = StandardScaler()
        X_train = scaler.fit_transform(X_train)
        X_test = scaler.transform(X_test)

        # LSTM expect a 3D matrix, so we reshape the training data with an extra
        # singular dimension to satisfy the dimension requirements
        X_train = np.reshape(X_train, (X_train.shape[0],1,X_train.shape[1]))
        X_test = np.reshape(X_test, (X_test.shape[0],1,X_test.shape[1]))

        self.x_testsets[k] = X_test
        self.y_testsets[k] = y_test


        if(hasattr(self,'model')):
            self.model.reset_states()
        else:
            self.model = Sequential()
            self.model.add(LSTM(batch_size_c, return_sequences=True, batch_size = batch_size_c,stateful=True,
                           input_shape=(timesteps, input_dim)))  # returns a sequence of vectors of dimension 32
            self.model.add(LSTM(batch_size_c, return_sequences=True))  # returns a sequence of vectors of dimension 32
            self.model.add(LSTM(batch_size_c))  # return a single vector of dimension 32
            self.model.add(Dense(32, activation='softmax'))
            self.model.add(Dense(32, activation='softmax'))
            self.model.add(Dense(6, activation='softmax'))
            opt = SGD(lr=0.003)
            self.model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])
        history = np.zeros(epochs)

        # States are reset at the end of each epoch, and reset at the start of a new timeseries. 
        for i in range(epochs):
            acc = self.model.fit(X_train, y_train, epochs=1, batch_size=batch_size_c, verbose=1, shuffle=False)
            history[i] = acc.history['acc'][0]
            self.model.reset_states()    
        # Plot accuracy evolution after each timeseries    
        plt.plot(range(0,epochs),history)
        plt.title('model accuracy')
        plt.ylabel('accuracy')
        plt.xlabel('epoch')
        plt.legend(['train'], loc='upper left')
        plt.show()

    X_test = np.concatenate([self.x_testsets[0],self.x_testsets[1],self.x_testsets[2]])
    y_test = np.concatenate([self.y_testsets[0],self.y_testsets[1],self.y_testsets[2]])

    # Ensure that the test dataset is divisible by batch size:
    deletedRows = 0
    while True: 
        if ((len(X_test)%batch_size_c == 0)):
            print("Test sets now divisible by batch size, deleted " + str(deletedRows) + " rows.")
            break
        X_test = np.delete(X_test,(len(X_test)-1),0)
        deletedRows = deletedRows + 1


    y_pred = self.model.predict(X_test,batch_size=batch_size_c)
    y_pred = (y_pred > 0.5)

Answer 1

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be tanh (or sigmoid) or ReLU (or any of the ReLU-type activations).

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $[0,1]$, so they model probabilities, but are not mutually exclusive.

OP reports in comments that after swapping hidden activations for ReLU and changing the readout activation to sigmoid, the oscillating accuracy phenomenon is still present. My conjecture is that this may be related to how the accuracy callback works; that is, it might implicitly assume that the target reflects one-hot classes, not many-hot.

The advice at What should I do when my neural network doesn't learn? is not specific to this problem, but the troubleshooting steps may be helpful.