Temporal Convolution Network for stage classification confusion [closed]

Using the TCN architecture provided in the repository here, in pytorch, I am trying to do multiclass classification this dataset. I have the following architecture: (will be abstracting some parts, that are the same as in the docs) This is the temporal block:

class TemporalBlock(nn.Module):
def __init__(self, n_inputs, n_outputs, kernel_size, stride, dilation, padding, dropout=0.2):
super(TemporalBlock, self).__init__()
self.conv1 = weight_norm(nn.Conv1d(n_inputs, n_outputs, kernel_size,
self.relu1 = nn.ReLU()
self.dropout1 = nn.Dropout(dropout)

# changed n_outputs to n_inputs, first param
self.conv2 = weight_norm(nn.Conv1d(n_outputs, n_outputs, kernel_size,
self.relu2 = nn.ReLU()
self.dropout2 = nn.Dropout(dropout)

self.net = nn.Sequential(self.conv1, self.chomp1, self.relu1, self.dropout1,
self.conv2, self.chomp2, self.relu2, self.dropout2)
self.downsample = nn.Conv1d(n_inputs, n_outputs, 1) if n_inputs != n_outputs else None
self.relu = nn.ReLU()
self.init_weights()


And this is the network:

class TemporalConvNet(nn.Module):
def __init__(self, num_inputs, num_channels, kernel_size=200, dropout=0.2):
super(TemporalConvNet, self).__init__()
layers = []
num_levels = len(num_channels)
for i in range(num_levels):
dilation_size = 2 ** i
in_channels = num_inputs if i == 0 else num_channels[i-1]
out_channels = num_channels[i]
layers += [TemporalBlock(in_channels, out_channels, kernel_size, stride=1, dilation=dilation_size,
padding=(kernel_size-1) * dilation_size, dropout=dropout)]
self.network = nn.Sequential(*layers)


This is the final module that I use using the aforementioned network:

class TCN(nn.Module):
def __init__(self, input_size, output_size, num_channels, kernel_size, dropout):
super(TCN, self).__init__()
self.tcn = TemporalConvNet(input_size, num_channels, kernel_size, dropout=dropout)
self.linear = nn.Linear(num_channels[-1], output_size)
self.sig = nn.Sigmoid()

def forward(self, x):
output = self.tcn(x)
output = self.linear(output[:, :, -1])
return self.sig(output).double()


And I initialize the model as follows:

    model = TCN(input_size, output_size, num_channels=[6]*30, kernel_size=125, dropout=0.25)


However, I have doubts regarding the TCN class, and whether I should use sigmoid as I have done in the output. This is how I train it:

size = 10
total_loss_s = 0
train_acc_s = 0
count = 0
perm = np.random.permutation(train_x.shape[0])
criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), betas=(0.9, 0.999), lr=1e-3, weight_decay=0)
for i in np.arange(0, train_x.shape[0], size):
x, y = train_x[perm[i:i + size]], train_y[perm[i:i + size]]
torch.cuda.empty_cache()
output = model(x)
train_loss = criterion(output, torch.max(y, 1)[1])
train_acc = multi_acc(output, torch.max(y, 1)[1])
train_loss.backward()
optimizer.step()
total_loss_s += train_loss.item()
train_acc_s += train_acc
count += output.size(0)
if i > 0 and i % 100 == 0:
cur_loss = total_loss_s / count
total_loss_s = 0.0
count = 0


Is there something that I have done wrong in the architecture or training step that someone else can see better? Any help is appreciated!! And the accuracy I get is low compared to when I only use CNN.

Its not advisable to use a sigmoid as activation function for your multiclass-classification scenario. If you would have only two classes sigmoid would be fine since the output space of sigmoid is $$\text{sigmoid}(x) \in (0,1)$$ and therefore a valid probability. Now you can use this probability as the probability for the positive class and $$1 - \text{sigmoid}(x)$$ as the probability for the negative class. This is however not possible if you have more than 2 classes, i.e. your not in a binary-classification setting anymore.
For this case you should use the softmax function as activation for your output layer. It scales all of your 4 outputs to valid probabilities. This is important since the loss of your network will be calculated using cross-entropy, which can only work correct if the sum of your output probabilities are valid, i.e. they sum up to $$1$$. This is ensured by the softmax function.