5
$\begingroup$

I'm a beginner in the field of Machine Learning and I'm currently trying to get my hands "dirty" for the first time with some code after completing a course in that field.

I'm using pyTorch to train a simple NN with one hidden layer. This is the code of my class:

class smallLayerNet(torch.nn.Module):

    def __init__(self, D_in, H, D_out):
        super(smallLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        sigmoid = torch.nn.Sigmoid()
        z1 = self.linear1(x)
        a1 = sigmoid(z1) # sigmoid activation
        z2 = self.linear2(a1)
        return z2

I'm using MSE for the loss function and Stochastic Gradient Descent for the optimization.

When running on 500 iterations on some random initialization I get a loss value of: 0.27523577213287354

However, if I remove the sigmoid activation, and the forward function looks as follows:

def forward(self, x):
        z1 = self.linear1(x)
        z2 = self.linear2(z1)
        return z2

after 500 iterations I get a loss value of 1.4318013788483519e-11 which is extremely better.

When I studied ML, I've learned that we want to use an activation function on the neurons, such as Sigmoid/ReLU/tanh. So - what am I missing here? Am I doing something wrong or am I wrong in my assumption?

Thanks!

Improve this question
$\endgroup$
6
  • $\begingroup$ You shouldn't use activation functions if your problem is linear. You didn't say anything about your data, so probably your data is better fitted by a linear model than a nonlinear one. $\endgroup$
    – Jakub Bartczuk
    Commented Oct 26, 2017 at 22:00
  • $\begingroup$ Also you can use some existing implementations for sanity checking your models - for this you can use appropriate regressors from scikit-learn (linear regression and MLPRegressor) $\endgroup$
    – Jakub Bartczuk
    Commented Oct 26, 2017 at 22:02
  • $\begingroup$ @JakubBartczuk Thanks for the comment. The data is "fake data" and I generate it randomly (the input and the output are both random numbers). Trying to use ReLU results in even worse results. $\endgroup$
    – Mickey
    Commented Oct 27, 2017 at 16:31
  • $\begingroup$ @Mickey If the data is random then how do you want to learn any kind of relationship between the two? If the signal is just noise then you will just fit to noise which won't result in anything "good". $\endgroup$
    – guy
    Commented Oct 27, 2017 at 16:47
  • 5
    $\begingroup$ @Mickey Most likely. Comparing models just using training loss values is not enough. You need to evaluate them on a dev/test set as well. $\endgroup$
    – guy
    Commented Oct 28, 2017 at 23:27

1 Answer 1

Reset to default
1
$\begingroup$

If you are trying to make a classification then sigmoid is necessary because you want to get a probability value. But if you are trying to make a scalar estimate then you would want not want to have a sigmoid since this would limit the output values btw 0 and 1.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Update: The first answer I wrote was for the activation functions for the fully connected layer or last layer. In your case I believe you would want to use another type of activation function such as ReLu. Sigmoid would limit the output of neurons btw 0 and 1 and i think this would cause problem in the calculations of gradients. $\endgroup$
    – BadSeed
    Commented Mar 17, 2018 at 18:20
  • $\begingroup$ I don't understand your answer. $\endgroup$
    – Michael R. Chernick
    Commented Mar 17, 2018 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.