I'm trying to make a LSTM model for detecting failures on a physical system, by supplying 27 features of sensor data. I've inputted three disjunct timeseries, each beginning with "normal" operational sensor readings before a failure occurs (each timeseries contains a new type of failure). Each timeseries has 10200 datapoints after splitting the sets in training and testing sets.

My main problem is that the accuracy gets continually degraded and suffers especially during a transition from one timeseries to the next (see attached figure). I'm curious if my implementation has methodical errors, and would be very grateful if someone would point any mistakes.

I've tried changing batch and epoch size, experimenting with different optimizers and layer combinations - but it still shows very strange transitional behaviour during the change from one timeseries to another.

Transition between one timeseries to the next Code:

    def model_initialize(self, datasets):
    num_datapoints = 0 

    for k in datasets:

        X = datasets[k].iloc[:, :-6] # Prediction variables
        y = datasets[k].iloc[:, -6:] # 

        num_datapoints = datasets[k].shape[0] + num_datapoints

        input_dim = 27
        # One timestep at a time, ensure that statefulness is enabled. 
        timesteps = 1
        batch_size_c = 12
        epochs = 100

        split_num = 0.7
        train_elements = 0
        test_elements = 0 

        if(self.variable_split == 0): 
            deletedRows = 0

            split = int(len(datasets[k])*split_num)
            train_elements = split
            test_elements = len(datasets[k])-split
            while True: 
                if ((train_elements%batch_size_c == 0) and (test_elements%batch_size_c == 0)):
                    print("Sets are now divisble by batch size - deleted " + str(deletedRows) +" rows")
                datasets[k].drop((datasets[k].shape[0]-1),inplace = True)
                split = int(len(datasets[k])*split_num)
                train_elements = split
                test_elements = len(datasets[k])-split
                deletedRows = deletedRows + 1

        X_train, X_test, y_train, y_test = X[:split], X[split:], y[:split], y[split:]

        # Feature Scaling

        scaler = StandardScaler()
        X_train = scaler.fit_transform(X_train)
        X_test = scaler.transform(X_test)

        # LSTM expect a 3D matrix, so we reshape the training data with an extra
        # singular dimension to satisfy the dimension requirements
        X_train = np.reshape(X_train, (X_train.shape[0],1,X_train.shape[1]))
        X_test = np.reshape(X_test, (X_test.shape[0],1,X_test.shape[1]))

        self.x_testsets[k] = X_test
        self.y_testsets[k] = y_test

            self.model = Sequential()
            self.model.add(LSTM(batch_size_c, return_sequences=True, batch_size = batch_size_c,stateful=True,
                           input_shape=(timesteps, input_dim)))  # returns a sequence of vectors of dimension 32
            self.model.add(LSTM(batch_size_c, return_sequences=True))  # returns a sequence of vectors of dimension 32
            self.model.add(LSTM(batch_size_c))  # return a single vector of dimension 32
            self.model.add(Dense(32, activation='softmax'))
            self.model.add(Dense(32, activation='softmax'))
            self.model.add(Dense(6, activation='softmax'))
            opt = SGD(lr=0.003)
            self.model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])
        history = np.zeros(epochs)

        # States are reset at the end of each epoch, and reset at the start of a new timeseries. 
        for i in range(epochs):
            acc = self.model.fit(X_train, y_train, epochs=1, batch_size=batch_size_c, verbose=1, shuffle=False)
            history[i] = acc.history['acc'][0]
        # Plot accuracy evolution after each timeseries    
        plt.title('model accuracy')
        plt.legend(['train'], loc='upper left')

    X_test = np.concatenate([self.x_testsets[0],self.x_testsets[1],self.x_testsets[2]])
    y_test = np.concatenate([self.y_testsets[0],self.y_testsets[1],self.y_testsets[2]])

    # Ensure that the test dataset is divisible by batch size:
    deletedRows = 0
    while True: 
        if ((len(X_test)%batch_size_c == 0)):
            print("Test sets now divisible by batch size, deleted " + str(deletedRows) + " rows.")
        X_test = np.delete(X_test,(len(X_test)-1),0)
        deletedRows = deletedRows + 1

    y_pred = self.model.predict(X_test,batch_size=batch_size_c)
    y_pred = (y_pred > 0.5)
  • $\begingroup$ Have you plotted what the loss looks like as training progresses? Does it steadily decline or does it also exhibit wide fluctuations? $\endgroup$ – Sycorax Mar 28 at 0:17

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be tanh (or sigmoid) or ReLU (or any of the ReLU-type activations).

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $[0,1]$, so they model probabilities, but are not mutually exclusive.

OP reports in comments that after swapping hidden activations for ReLU and changing the readout activation to sigmoid, the oscillating accuracy phenomenon is still present. My conjecture is that this may be related to how the accuracy callback works; that is, it might implicitly assume that the target reflects one-hot classes, not many-hot.

The advice at What should I do when my neural network doesn't learn? is not specific to this problem, but the troubleshooting steps may be helpful.

  • $\begingroup$ @Splotsmeister please see my edit $\endgroup$ – Sycorax Mar 27 at 18:33
  • $\begingroup$ Very sorry for deleting my comment, I realized what you wrote about the incompatability of the softmax function as you answered me. I changed the last layer to a sigmoid and it correctly does multilabeling now. However, the accuracy degradation is still present when a new time series is fed to the model, so the fundamental problem is still present it seems. $\endgroup$ – Splotsmeister Mar 27 at 18:36
  • $\begingroup$ Usually, accuracy is computed for a "one-hot" classification task. Are you sure that the accuracy metric that is used here is suitable for "many-hot" classification? $\endgroup$ – Sycorax Mar 27 at 18:39
  • $\begingroup$ I'm not sure, I shall look into it. Does the metric given during the compile line impact the actual training of the model, or does it only serve as a feedback for the user about the performance of the model? $\endgroup$ – Splotsmeister Mar 27 at 18:42
  • $\begingroup$ Only the loss function is used for back-propagation (model training). The metric just gives the user feedback on how well the model is doing according to that metric. This might be worth reading over: pyimagesearch.com/2018/05/07/… $\endgroup$ – Sycorax Mar 27 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.