If I have data generated from an underlying process like this one: $y = a + b x_1 + c x_2 + d x_1 x_2 + noise$

How would neural networks represent the interaction term between $x_1$ and $x_2$? Is there a special type of unit that can output (a linear combination of) the interactions between its inputs? Or does a network have to learn to approximate multiplication just to express the $x_1 x_2$ interaction term?

I know we could simply include $x_1 x_2$ as an additional input (i.e., basis expansion), but I'm trying to understand if there is any way to avoid the exponential blow up in inputs with the dimensionality of $x$, and instead automate the learning of interactions.

  • Yes, it has to learn to approximate it using whatever units it has. – Neil G May 14 '17 at 18:33

The idea is that Neural Networks don't need to be supplied with handcrafted features. If you give someone else your generated data, without telling them the underlying process, they will have direct access to just x1, x2, and y. Feeding x1 and x2 into the network and instructing it to predict y should make the network model the underlying process. However, it might need a lot of examples before it is accurately able to do so.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.