0
$\begingroup$

I am getting started with very simple Neural Networks/Multilayer Perceptrons. I successfully classified the XOR problem, but I wanted to explore so I created a grid such as enter image description here.

I used Tensorflow code and NN model from this article.

Network Details: 2 inputs, 1 hidden layer with 2 neurons, and 1 output layer with 1 neuron. Sigmoid activation function is used. The cost function is the "average over all the training examples" according to the link: [ y * log(y_hat) - (1 - y) * log( 1-y_hat ) ]. Gradient Descent with learning rate of 0.01 is used to train the algorithm. Weights initialized from uniform distribution between -1 and 1, biases initialized to 0.

I successfully trained the algorithm on the grid: enter image description here

Then, I held out 1 point but was unable to converge the algorithm: enter image description here

Next, I held out the 2 corner points and was able to converge the algorithm this time: enter image description here

Questions: (1) Why is the classifier not working very well in the 15-point case? It seems to be satisfied with classifying that corner and disregards the 2 points in the middle. (2) Is there a technique I can use to sort of "nudge it" towards the right solution? Different initialization?

$\endgroup$
0
$\begingroup$

I was able to get the NN to converge correctly by standardizing the data points using sci-kit learn's StandardScaler function. Notice that my original grids were plotted between [0,1] on both axes. The StandardScaler rescales it to be on the interval of [-1.5,1.5].

enter image description here

Why does this solve the problem you may ask? Well, I initialized the biases to 0. According to this link, the biases determine the distance of the initial decision boundary to the origin. By restructuring the data to be around the origin, the NN had to do less work to get those decision boundaries in the right places.

Hope this helps anyone interested in understanding this problem!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.