# What exactly happens when I do a feature cross?

I was going through a machine learning course and they talked about combining various features to create synthetic feature to take care of non linear data. For eg in the below picture I didn't do any feature crossing and the model didn't fit:

But if I do some feature crossing and create/activate features $$x_1^2$$, $$x_2^2$$ and $$x_1x_2$$ I get this:

The model fits now. But why? What exactly does feature crossing do that enables a model to fit non linear data?

Can some one please help me understand it?

## 1 Answer

Your data is not linearly separable in the original space.

But it seems like it actually is separable with a circle/ellipse (let's say it's inside a circle to simplify the problem): it seems reasonable to have hypothesis that, for some $c$ if $x^2 + y^2< c$ then a point is blue.

That means that if you use $x^2, y^2$ as features, you can fit a linear classifier to these data points and actually separate the classes linearly.

• Ok, so if I understand it right it basically transforms the space so that in the new space the inner circle is at a higher altitude than the outer circle or vice versa(like a hill with the blue points at top) such that our linear classifier can separate it with a single slice. Is this thinking correct? – Lord_Rhaziel Mar 4 '18 at 10:36
• If you take just $x^2, y^2$ then this is correct reasoning. But I don't know if it works exactly that way in this simulation - I don't see if you disabled the original features. – Jakub Bartczuk Mar 4 '18 at 10:48
• Ok, I just tried it in the simulation with original features disabled and only $x_1^2$, $x_2^2$ & $x_1x_2$ enabled. It still fits, better in fact. So should I conclude that original features do not matter now? Why do they not matter anymore? Is it because the information they used to carry are redundant now? – Lord_Rhaziel Mar 4 '18 at 13:11