Using the TCN architecture provided in the repository here, in pytorch, I am trying to do multiclass classification this dataset. I have the following architecture: (will be abstracting some parts, that are the same as in the docs) This is the temporal block:

class TemporalBlock(nn.Module):
def __init__(self, n_inputs, n_outputs, kernel_size, stride, dilation, padding, dropout=0.2):
    super(TemporalBlock, self).__init__()
    self.conv1 = weight_norm(nn.Conv1d(n_inputs, n_outputs, kernel_size,
                                       stride=stride, padding=padding, dilation=dilation))
    self.chomp1 = Chomp1d(padding)
    self.relu1 = nn.ReLU()
    self.dropout1 = nn.Dropout(dropout)

    # changed n_outputs to n_inputs, first param
    self.conv2 = weight_norm(nn.Conv1d(n_outputs, n_outputs, kernel_size,
                                       stride=stride, padding=padding, dilation=dilation))
    self.chomp2 = Chomp1d(padding)
    self.relu2 = nn.ReLU()
    self.dropout2 = nn.Dropout(dropout)

    self.net = nn.Sequential(self.conv1, self.chomp1, self.relu1, self.dropout1,
                             self.conv2, self.chomp2, self.relu2, self.dropout2)
    self.downsample = nn.Conv1d(n_inputs, n_outputs, 1) if n_inputs != n_outputs else None
    self.relu = nn.ReLU()

And this is the network:

class TemporalConvNet(nn.Module):
def __init__(self, num_inputs, num_channels, kernel_size=200, dropout=0.2):
    super(TemporalConvNet, self).__init__()
    layers = []
    num_levels = len(num_channels)
    for i in range(num_levels):
        dilation_size = 2 ** i
        in_channels = num_inputs if i == 0 else num_channels[i-1]
        out_channels = num_channels[i]
        layers += [TemporalBlock(in_channels, out_channels, kernel_size, stride=1, dilation=dilation_size,
                                 padding=(kernel_size-1) * dilation_size, dropout=dropout)]
    self.network = nn.Sequential(*layers)

This is the final module that I use using the aforementioned network:

class TCN(nn.Module):
def __init__(self, input_size, output_size, num_channels, kernel_size, dropout):
    super(TCN, self).__init__()
    self.tcn = TemporalConvNet(input_size, num_channels, kernel_size, dropout=dropout)
    self.linear = nn.Linear(num_channels[-1], output_size)
    self.sig = nn.Sigmoid()

def forward(self, x):
    output = self.tcn(x)
    output = self.linear(output[:, :, -1])
    return self.sig(output).double()

And I initialize the model as follows:

    model = TCN(input_size, output_size, num_channels=[6]*30, kernel_size=125, dropout=0.25)

However, I have doubts regarding the TCN class, and whether I should use sigmoid as I have done in the output. This is how I train it:

size = 10
    total_loss_s = 0
    train_acc_s = 0
    count = 0
    perm = np.random.permutation(train_x.shape[0])
    criterion = torch.nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), betas=(0.9, 0.999), lr=1e-3, weight_decay=0)
    for i in np.arange(0, train_x.shape[0], size):
        x, y = train_x[perm[i:i + size]], train_y[perm[i:i + size]]
        output = model(x)
        train_loss = criterion(output, torch.max(y, 1)[1])
        train_acc = multi_acc(output, torch.max(y, 1)[1])
        total_loss_s += train_loss.item()
        train_acc_s += train_acc
        count += output.size(0)
        if i > 0 and i % 100 == 0:
            cur_loss = total_loss_s / count
            total_loss_s = 0.0
            count = 0

Is there something that I have done wrong in the architecture or training step that someone else can see better? Any help is appreciated!! And the accuracy I get is low compared to when I only use CNN.


Its not advisable to use a sigmoid as activation function for your multiclass-classification scenario. If you would have only two classes sigmoid would be fine since the output space of sigmoid is $\text{sigmoid}(x) \in (0,1)$ and therefore a valid probability. Now you can use this probability as the probability for the positive class and $1 - \text{sigmoid}(x)$ as the probability for the negative class. This is however not possible if you have more than 2 classes, i.e. your not in a binary-classification setting anymore.

For this case you should use the softmax function as activation for your output layer. It scales all of your 4 outputs to valid probabilities. This is important since the loss of your network will be calculated using cross-entropy, which can only work correct if the sum of your output probabilities are valid, i.e. they sum up to $1$. This is ensured by the softmax function.

| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.