# LSTM - Multiple Time Series, degrading accuracy

I'm trying to make a LSTM model for detecting failures on a physical system, by supplying 27 features of sensor data. I've inputted three disjunct timeseries, each beginning with "normal" operational sensor readings before a failure occurs (each timeseries contains a new type of failure). Each timeseries has 10200 datapoints after splitting the sets in training and testing sets.

My main problem is that the accuracy gets continually degraded and suffers especially during a transition from one timeseries to the next (see attached figure). I'm curious if my implementation has methodical errors, and would be very grateful if someone would point any mistakes.

I've tried changing batch and epoch size, experimenting with different optimizers and layer combinations - but it still shows very strange transitional behaviour during the change from one timeseries to another.

Code:

    def model_initialize(self, datasets):
num_datapoints = 0

for k in datasets:

X = datasets[k].iloc[:, :-6] # Prediction variables
y = datasets[k].iloc[:, -6:] #

num_datapoints = datasets[k].shape[0] + num_datapoints

input_dim = 27
# One timestep at a time, ensure that statefulness is enabled.
timesteps = 1
batch_size_c = 12
epochs = 100

split_num = 0.7
train_elements = 0
test_elements = 0

if(self.variable_split == 0):
deletedRows = 0

split = int(len(datasets[k])*split_num)
train_elements = split
test_elements = len(datasets[k])-split
while True:
if ((train_elements%batch_size_c == 0) and (test_elements%batch_size_c == 0)):
print("Sets are now divisble by batch size - deleted " + str(deletedRows) +" rows")
break
datasets[k].drop((datasets[k].shape[0]-1),inplace = True)
split = int(len(datasets[k])*split_num)
train_elements = split
test_elements = len(datasets[k])-split
deletedRows = deletedRows + 1

X_train, X_test, y_train, y_test = X[:split], X[split:], y[:split], y[split:]

# Feature Scaling

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# LSTM expect a 3D matrix, so we reshape the training data with an extra
# singular dimension to satisfy the dimension requirements
X_train = np.reshape(X_train, (X_train.shape[0],1,X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0],1,X_test.shape[1]))

self.x_testsets[k] = X_test
self.y_testsets[k] = y_test

if(hasattr(self,'model')):
self.model.reset_states()
else:
self.model = Sequential()
input_shape=(timesteps, input_dim)))  # returns a sequence of vectors of dimension 32
self.model.add(LSTM(batch_size_c, return_sequences=True))  # returns a sequence of vectors of dimension 32
self.model.add(LSTM(batch_size_c))  # return a single vector of dimension 32
opt = SGD(lr=0.003)
self.model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])
history = np.zeros(epochs)

# States are reset at the end of each epoch, and reset at the start of a new timeseries.
for i in range(epochs):
acc = self.model.fit(X_train, y_train, epochs=1, batch_size=batch_size_c, verbose=1, shuffle=False)
history[i] = acc.history['acc'][0]
self.model.reset_states()
# Plot accuracy evolution after each timeseries
plt.plot(range(0,epochs),history)
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train'], loc='upper left')
plt.show()

X_test = np.concatenate([self.x_testsets[0],self.x_testsets[1],self.x_testsets[2]])
y_test = np.concatenate([self.y_testsets[0],self.y_testsets[1],self.y_testsets[2]])

# Ensure that the test dataset is divisible by batch size:
deletedRows = 0
while True:
if ((len(X_test)%batch_size_c == 0)):
print("Test sets now divisible by batch size, deleted " + str(deletedRows) + " rows.")
break
X_test = np.delete(X_test,(len(X_test)-1),0)
deletedRows = deletedRows + 1

y_pred = self.model.predict(X_test,batch_size=batch_size_c)
y_pred = (y_pred > 0.5)

• Have you plotted what the loss looks like as training progresses? Does it steadily decline or does it also exhibit wide fluctuations? – Sycorax Mar 28 at 0:17

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be tanh (or sigmoid) or ReLU (or any of the ReLU-type activations).

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $$[0,1]$$, so they model probabilities, but are not mutually exclusive.

OP reports in comments that after swapping hidden activations for ReLU and changing the readout activation to sigmoid, the oscillating accuracy phenomenon is still present. My conjecture is that this may be related to how the accuracy callback works; that is, it might implicitly assume that the target reflects one-hot classes, not many-hot.

The advice at What should I do when my neural network doesn't learn? is not specific to this problem, but the troubleshooting steps may be helpful.

• @Splotsmeister please see my edit – Sycorax Mar 27 at 18:33
• Very sorry for deleting my comment, I realized what you wrote about the incompatability of the softmax function as you answered me. I changed the last layer to a sigmoid and it correctly does multilabeling now. However, the accuracy degradation is still present when a new time series is fed to the model, so the fundamental problem is still present it seems. – Splotsmeister Mar 27 at 18:36
• Usually, accuracy is computed for a "one-hot" classification task. Are you sure that the accuracy metric that is used here is suitable for "many-hot" classification? – Sycorax Mar 27 at 18:39
• I'm not sure, I shall look into it. Does the metric given during the compile line impact the actual training of the model, or does it only serve as a feedback for the user about the performance of the model? – Splotsmeister Mar 27 at 18:42
• Only the loss function is used for back-propagation (model training). The metric just gives the user feedback on how well the model is doing according to that metric. This might be worth reading over: pyimagesearch.com/2018/05/07/… – Sycorax Mar 27 at 18:43