# Sigmoid activation hurts training a NN on pyTorch

I'm a beginner in the field of Machine Learning and I'm currently trying to get my hands "dirty" for the first time with some code after completing a course in that field.

I'm using pyTorch to train a simple NN with one hidden layer. This is the code of my class:

class smallLayerNet(torch.nn.Module):

def __init__(self, D_in, H, D_out):
super(smallLayerNet, self).__init__()
self.linear1 = torch.nn.Linear(D_in, H)
self.linear2 = torch.nn.Linear(H, D_out)

def forward(self, x):
sigmoid = torch.nn.Sigmoid()
z1 = self.linear1(x)
a1 = sigmoid(z1) # sigmoid activation
z2 = self.linear2(a1)
return z2


I'm using MSE for the loss function and Stochastic Gradient Descent for the optimization.

When running on 500 iterations on some random initialization I get a loss value of: 0.27523577213287354

However, if I remove the sigmoid activation, and the forward function looks as follows:

def forward(self, x):
z1 = self.linear1(x)
z2 = self.linear2(z1)
return z2


after 500 iterations I get a loss value of 1.4318013788483519e-11 which is extremely better.

When I studied ML, I've learned that we want to use an activation function on the neurons, such as Sigmoid/ReLU/tanh. So - what am I missing here? Am I doing something wrong or am I wrong in my assumption?

Thanks!

• You shouldn't use activation functions if your problem is linear. You didn't say anything about your data, so probably your data is better fitted by a linear model than a nonlinear one. – Jakub Bartczuk Oct 26 '17 at 22:00
• Also you can use some existing implementations for sanity checking your models - for this you can use appropriate regressors from scikit-learn (linear regression and MLPRegressor) – Jakub Bartczuk Oct 26 '17 at 22:02
• @JakubBartczuk Thanks for the comment. The data is "fake data" and I generate it randomly (the input and the output are both random numbers). Trying to use ReLU results in even worse results. – Mickey Oct 27 '17 at 16:31
• @Mickey If the data is random then how do you want to learn any kind of relationship between the two? If the signal is just noise then you will just fit to noise which won't result in anything "good". – guy Oct 27 '17 at 16:47
• @Mickey Most likely. Comparing models just using training loss values is not enough. You need to evaluate them on a dev/test set as well. – guy Oct 28 '17 at 23:27