Binary Classification problem without using Sigmoid? I'm have a network where I am using BCE for loss and a sigmoid layer as the last layer's activation function. If I wanted to remove sigmoid, and declare negatives to be one class and positives to be another class(instead of the traditional .5 being the marker between the classes), would that work? Would the network be able to adapt properly? I'm trying to toy with the idea of making a smaller network and I'm curious if this idea holds any weight or if I'm fundamentally misunderstanding something.
 A: There's nothing invalid about defining a threshold for classification.
Such procedure, however, is non-differentiable, so therefore you won't be able to train that network using that setup (see that you can adopt it for an already trained network if it suits your task).
This is simply because the output becomes
$$\mathcal F(\mathcal N (X)) = \mathbb 1_{\mathcal N (X)>0.5}= \begin{cases}0, \quad \text{if} \quad \mathcal N (X)\leq 0.5\\1, \quad \text{if} \quad \mathcal N (X)>0.5\end{cases}$$
And the derivative regarding the inputs of $\mathcal F$ are zero everywhere except at the threshold, where it is undefined.
If you need to train such a network and don't actually require the probabilities for anything afterwards, just the classification, you can ditch the sigmoid and instead adopt a hinge-loss, à la SVMs.
This will give you a differentiable dichotomized output.
A: The sigmoid function $\sigma(x)=\frac{1}{1 + \exp(-x)}$ has the property $\sigma(0)=0.5$. The sigmoid function is also monotonic increasing. Together, these facts imply that $\sigma(x) < 0$ whenever $x < 0$ and $\sigma(x)> 0$ whenever $x>0$.
If all you're interested in doing is classifying according to the sign of some part of the network, then you can train with the sigmoid activation and the ordinary cross-entropy loss, but predict using the value of $x$ instead of $\sigma(x)$. In other words, just skip $\sigma$ for making predictions. Due to the connection to logistic regression, this is the same as making predictions using the logit scale instead of the probability scale.
