How to achieve a nonlinear decision boundary?

Question

What would be the architecture of the neural net that would produce the following nonlinear decision boundary?

Will the hidden layer compute some nonlinear combinations of inputs? or it will create several linear decision boundaries by computing linear combinations of inputs and then produce a convex region?

Answer 1

This is a simple dataset, even though it isn't linearly separable. A Multilayer perceptron is able to correctly classify this dataset.

The minimal architecture necessary to correctly classify this dataset requires 2 neurons for the input layer, 3 neurons in the hidden layer and 1 neuron in the output.

The three neurons in the hidden layer will learn to disentangle the data and disperse them in a 3-dimenional space such that they will become linearly separable in this new space.

You can evaluate how the learning varies depending on the activation function you use, especially for the hidden layer. Try to compare for example sigmoid, ReLu and tanh activation functions.

Here there's a fantastic explanation of what happens in the hidden layer when learning to classify this exact same dataset.

Answer 2

Just out of curiosity, I want to add that the above data that you showcased is linearly separable if you change the feature space from Cartesian coordinate system to polar coordinate system. Once converted to the polar coordinate system any simple classifier would do the job.

Sample visualization of similar data:

Answer 3

this should help. you can try out various parameters, hit the play button and see for yourself what works.

for a theoretical breakdown of the problem you can refer this

Will the hidden layer compute some nonlinear combinations of inputs?

I didn't fully understand the non linear 'combination' part though.