Return to Answer

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be sigmoid or relu.

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $[0,1]$, so they model probabilities, but are not mutually exclusive.

OP reports in comments that after swapping hidden activations for ReLU and changing the readout activation to sigmoid, the oscillating accuracy phenomenon is still present. My conjecture is that this may be related to how the accuracy callback works; that is, it might implicitly assume that the target reflects one-hot classes, not many-hot.

The advice at What should I do when my neural network doesn't learn? is not specific to this problem, but the troubleshooting steps may be helpful.

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be tanh (or sigmoid) or ReLU (or any of the ReLU-type activations).

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $[0,1]$, so they model probabilities, but are not mutually exclusive.

OP reports in comments that after swapping hidden activations for ReLU and changing the readout activation to sigmoid, the oscillating accuracy phenomenon is still present. My conjecture is that this may be related to how the accuracy callback works; that is, it might implicitly assume that the target reflects one-hot classes, not many-hot.

The advice at What should I do when my neural network doesn't learn? is not specific to this problem, but the troubleshooting steps may be helpful.

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be sigmoid or relu.

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $[0,1]$, so they model probabilities, but are not mutually exclusive.

The advice at What should I do when my neural network doesn't learn? is not specific to this problem, but the troubleshooting steps may be helpful.

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be sigmoid or relu.

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $[0,1]$, so they model probabilities, but are not mutually exclusive.

OP reports in comments that after swapping hidden activations for ReLU and changing the readout activation to sigmoid, the oscillating accuracy phenomenon is still present. My conjecture is that this may be related to how the accuracy callback works; that is, it might implicitly assume that the target reflects one-hot classes, not many-hot.

The advice at What should I do when my neural network doesn't learn? is not specific to this problem, but the troubleshooting steps may be helpful.

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be sigmoid or relu.

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $[0,1]$, so they model probabilities, but are not mutually exclusive.

It seems strange to use softmax for the hidden activations of your dense layer. A typical choice would be sigmoid or relu.

In a comment, OP wrote

I updated to a Relu->Relu->Softmax setup now, and the accuracy plots are less jumpy/noisy, but it still presents the same degradation upon being fed new timeseries. Additionally it now seems to only classify one of 6 possible output hot-encoded classes as True, where it would correctly flag multiple classes previously - albeit with bad accuracy. Are some of these activation functions incompatible with multi-class problems where several classes may be true simultaneously?

Softmax outputs a probability vector, so its elements are non-negative and sum to 1. This implies that softmax is a good choice to model mutually-exclusive outcomes (a sequence can be class A or B but not both). If your outcomes are not mutually exclusive (a sequence could be class A or class B or class A and B), then you should use a different activation on the final layer, such as a sigmoid activation. Sigmoid activations have outputs in $[0,1]$, so they model probabilities, but are not mutually exclusive.

The advice at What should I do when my neural network doesn't learn? is not specific to this problem, but the troubleshooting steps may be helpful.