I recently finished my implementation of a feedforward multilayer ANN in Julia. I train it using basic gradient descent with no additions (no regularization, no momentum, no decay, no anything, just vanilla Gradient Descent, computing the gradient through backpropagation). The weights and biases are initialized sampling a standard normal distribution.
What surprises me is that, with my implementation, I cannot make a single sigmoid neuron classify with 100% accuracy a linearly separable dataset, even when training the neuron against that very same set for 5000 iterations or more.
The training set consists of a set of $n = 1000$ randomly generated input vectors $x_1, x_2, ..., x_n$, such that, for $k = 1, 2, ..., n$,
$$x_k = (x_{k1}, x_{k2}, ..., x_{k5}) \qquad x_{ki} \sim \mathcal{N}(0, 1)$$
There is a corresponding set of labels $y_1, y_2, ..., y_n$, with $y_k \in \{0, 1\}$ for $k = 1, 2, ..., n$, such that
$$y_k = 1 \;\iff\; \sum_i x_{ki} > 0$$
If the output of the network, rounded to the nearest integer, given $x_k$ as input, is equal to $y_k$, then the network has correctly classified $x_k$.
Training the network with this set, and testing against the same set, I can achieve an accuracy of 0.997 tops. Not even with 5000 iterations, however, I can get the neuron to classify the whole set correctly. I tried with different learning rates, but there are always some samples that the neuron will simply not classify well (by a small margin, maybe, like 0.02, but it simply does not).
Questions: Does this result make sense? Cannot a sigmoid neuron, trained by means of the classical Gradient Descent, classify all the elements of such a set correctly for most initializations? Could there be some kind of problem with my implementation? Numerical issues maybe? Is there some kind of bound on the accuracy that can be achieved given the learning rate, and the margin between the two regions to be separated?
