This is not a problem:solution scenario insofar that I am not attempting to find a way to improve the model, merely find a reason for its behaviour.
Model using LSTM has accuracy of about 84-86% while LSTM featuring a CNN layer (CNN followed by LSTM) has accuracy of about 82% on the same dataset.
Received wisdom seems to suggest that adding a CNN layer would improve performance. The hyperparameters of the CNN was determined through Bayesian optimization, with quite a broad search space (kernel size, activation function, number of convolutional filters, etc.). 82% seems to be about the best it can do, with an ideal setup.
Are there are any reasons which might help explain the slight drop in performance when adding a convolution layer to a RNN? The data in this case are word vectors generated from a noisy source (think Twitter).