I recently finished my implementation of a feedforward multilayer ANN in Julia. I train it using basic gradient descent with no additions (no regularization, no momentum, no decay, no anything, just vanilla Gradient Descent, computing the gradient through backpropagation). The weights and biases are initialized sampling a standard normal distribution.

What surprises me is that, with my implementation, I cannot make a single sigmoid neuron classify with 100% accuracy a linearly separable dataset, even when training the neuron against that very same set for 5000 iterations or more.

The training set consists of a set of $n = 1000$ randomly generated input vectors $x_1, x_2, ..., x_n$, such that, for $k = 1, 2, ..., n$,

$$x_k = (x_{k1}, x_{k2}, ..., x_{k5}) \qquad x_{ki} \sim \mathcal{N}(0, 1)$$

There is a corresponding set of labels $y_1, y_2, ..., y_n$, with $y_k \in \{0, 1\}$ for $k = 1, 2, ..., n$, such that

$$y_k = 1 \;\iff\; \sum_i x_{ki} > 0$$

If the output of the network, rounded to the nearest integer, given $x_k$ as input, is equal to $y_k$, then the network has correctly classified $x_k$.

Training the network with this set, and testing against the same set, I can achieve an accuracy of 0.997 tops. Not even with 5000 iterations, however, I can get the neuron to classify the whole set correctly. I tried with different learning rates, but there are always some samples that the neuron will simply not classify well (by a small margin, maybe, like 0.02, but it simply does not).

Questions: Does this result make sense? Cannot a sigmoid neuron, trained by means of the classical Gradient Descent, classify all the elements of such a set correctly for most initializations? Could there be some kind of problem with my implementation? Numerical issues maybe? Is there some kind of bound on the accuracy that can be achieved given the learning rate, and the margin between the two regions to be separated?

  • $\begingroup$ Try the experiment with another library with the same parameters and see if it works. $\endgroup$ – Chris Mar 29 '16 at 14:18

In general it is not possible. This is just a convex optimization problem, which (given reasonably small learning ratę) will be solved even by vanilla gradient descent. Obviously smaller the margin smaller the learning rate required for convergence, but for such simple problem it should not matter. This is also rather impossible to be a numerical issue. This look like a simple bug. Post the question, with code to SO

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.