I was going through a machine learning course and they talked about combining various features to create synthetic feature to take care of non linear data. For eg in the below picture I didn't do any feature crossing and the model didn't fit: no feature crossing

But if I do some feature crossing and create/activate features $x_1^2$, $x_2^2$ and $x_1x_2$ I get this:Fits with feature crossing

The model fits now. But why? What exactly does feature crossing do that enables a model to fit non linear data?

Can some one please help me understand it?


Your data is not linearly separable in the original space.

But it seems like it actually is separable with a circle/ellipse (let's say it's inside a circle to simplify the problem): it seems reasonable to have hypothesis that, for some $c$ if $x^2 + y^2< c$ then a point is blue.

That means that if you use $x^2, y^2$ as features, you can fit a linear classifier to these data points and actually separate the classes linearly.

  • $\begingroup$ Ok, so if I understand it right it basically transforms the space so that in the new space the inner circle is at a higher altitude than the outer circle or vice versa(like a hill with the blue points at top) such that our linear classifier can separate it with a single slice. Is this thinking correct? $\endgroup$ – Lord_Rhaziel Mar 4 '18 at 10:36
  • $\begingroup$ If you take just $x^2, y^2$ then this is correct reasoning. But I don't know if it works exactly that way in this simulation - I don't see if you disabled the original features. $\endgroup$ – Jakub Bartczuk Mar 4 '18 at 10:48
  • $\begingroup$ Ok, I just tried it in the simulation with original features disabled and only $x_1^2$, $x_2^2$ & $x_1x_2$ enabled. It still fits, better in fact. So should I conclude that original features do not matter now? Why do they not matter anymore? Is it because the information they used to carry are redundant now? $\endgroup$ – Lord_Rhaziel Mar 4 '18 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.