I'm a beginner in the field of Machine Learning and I'm currently trying to get my hands "dirty" for the first time with some code after completing a course in that field.

I'm using pyTorch to train a simple NN with one hidden layer. This is the code of my class:

class smallLayerNet(torch.nn.Module):

    def __init__(self, D_in, H, D_out):
        super(smallLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        sigmoid = torch.nn.Sigmoid()
        z1 = self.linear1(x)
        a1 = sigmoid(z1) # sigmoid activation
        z2 = self.linear2(a1)
        return z2

I'm using MSE for the loss function and Stochastic Gradient Descent for the optimization.

When running on 500 iterations on some random initialization I get a loss value of: 0.27523577213287354

However, if I remove the sigmoid activation, and the forward function looks as follows:

def forward(self, x):
        z1 = self.linear1(x)
        z2 = self.linear2(z1)
        return z2

after 500 iterations I get a loss value of 1.4318013788483519e-11 which is extremely better.

When I studied ML, I've learned that we want to use an activation function on the neurons, such as Sigmoid/ReLU/tanh. So - what am I missing here? Am I doing something wrong or am I wrong in my assumption?


  • $\begingroup$ You shouldn't use activation functions if your problem is linear. You didn't say anything about your data, so probably your data is better fitted by a linear model than a nonlinear one. $\endgroup$ – Jakub Bartczuk Oct 26 '17 at 22:00
  • $\begingroup$ Also you can use some existing implementations for sanity checking your models - for this you can use appropriate regressors from scikit-learn (linear regression and MLPRegressor) $\endgroup$ – Jakub Bartczuk Oct 26 '17 at 22:02
  • $\begingroup$ @JakubBartczuk Thanks for the comment. The data is "fake data" and I generate it randomly (the input and the output are both random numbers). Trying to use ReLU results in even worse results. $\endgroup$ – Mickey Oct 27 '17 at 16:31
  • $\begingroup$ @Mickey If the data is random then how do you want to learn any kind of relationship between the two? If the signal is just noise then you will just fit to noise which won't result in anything "good". $\endgroup$ – guy Oct 27 '17 at 16:47
  • 5
    $\begingroup$ @Mickey Most likely. Comparing models just using training loss values is not enough. You need to evaluate them on a dev/test set as well. $\endgroup$ – guy Oct 28 '17 at 23:27

If you are trying to make a classification then sigmoid is necessary because you want to get a probability value. But if you are trying to make a scalar estimate then you would want not want to have a sigmoid since this would limit the output values btw 0 and 1.

| cite | improve this answer | |
  • $\begingroup$ Update: The first answer I wrote was for the activation functions for the fully connected layer or last layer. In your case I believe you would want to use another type of activation function such as ReLu. Sigmoid would limit the output of neurons btw 0 and 1 and i think this would cause problem in the calculations of gradients. $\endgroup$ – BadSeed Mar 17 '18 at 18:20
  • $\begingroup$ I don't understand your answer. $\endgroup$ – Michael R. Chernick Mar 17 '18 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.