# Sigmoid activation hurts training a NN on pyTorch

I'm a beginner in the field of Machine Learning and I'm currently trying to get my hands "dirty" for the first time with some code after completing a course in that field.

I'm using pyTorch to train a simple NN with one hidden layer. This is the code of my class:

class smallLayerNet(torch.nn.Module):

def __init__(self, D_in, H, D_out):
super(smallLayerNet, self).__init__()
self.linear1 = torch.nn.Linear(D_in, H)
self.linear2 = torch.nn.Linear(H, D_out)

def forward(self, x):
sigmoid = torch.nn.Sigmoid()
z1 = self.linear1(x)
a1 = sigmoid(z1) # sigmoid activation
z2 = self.linear2(a1)
return z2


I'm using MSE for the loss function and Stochastic Gradient Descent for the optimization.

When running on 500 iterations on some random initialization I get a loss value of: 0.27523577213287354

However, if I remove the sigmoid activation, and the forward function looks as follows:

def forward(self, x):
z1 = self.linear1(x)
z2 = self.linear2(z1)
return z2


after 500 iterations I get a loss value of 1.4318013788483519e-11 which is extremely better.

When I studied ML, I've learned that we want to use an activation function on the neurons, such as Sigmoid/ReLU/tanh. So - what am I missing here? Am I doing something wrong or am I wrong in my assumption?

Thanks!

• You shouldn't use activation functions if your problem is linear. You didn't say anything about your data, so probably your data is better fitted by a linear model than a nonlinear one. Commented Oct 26, 2017 at 22:00
• Also you can use some existing implementations for sanity checking your models - for this you can use appropriate regressors from scikit-learn (linear regression and MLPRegressor) Commented Oct 26, 2017 at 22:02
• @JakubBartczuk Thanks for the comment. The data is "fake data" and I generate it randomly (the input and the output are both random numbers). Trying to use ReLU results in even worse results. Commented Oct 27, 2017 at 16:31
• @Mickey If the data is random then how do you want to learn any kind of relationship between the two? If the signal is just noise then you will just fit to noise which won't result in anything "good".
– guy
Commented Oct 27, 2017 at 16:47
• @Mickey Most likely. Comparing models just using training loss values is not enough. You need to evaluate them on a dev/test set as well.
– guy
Commented Oct 28, 2017 at 23:27