13
$\begingroup$

If I have data generated from an underlying process like this one: $y = a + b x_1 + c x_2 + d x_1 x_2 + noise$

How would neural networks represent the interaction term between $x_1$ and $x_2$? Is there a special type of unit that can output (a linear combination of) the interactions between its inputs? Or does a network have to learn to approximate multiplication just to express the $x_1 x_2$ interaction term?

I know we could simply include $x_1 x_2$ as an additional input (i.e., basis expansion), but I'm trying to understand if there is any way to avoid the exponential blow up in inputs with the dimensionality of $x$, and instead automate the learning of interactions.

$\endgroup$
1
  • $\begingroup$ Yes, it has to learn to approximate it using whatever units it has. $\endgroup$
    – Neil G
    May 14, 2017 at 18:33

2 Answers 2

2
$\begingroup$

The idea is that Neural Networks don't need to be supplied with handcrafted features. If you give someone else your generated data, without telling them the underlying process, they will have direct access to just x1, x2, and y. Feeding x1 and x2 into the network and instructing it to predict y should make the network model the underlying process. However, it might need a lot of examples before it is accurately able to do so.

$\endgroup$
3
  • 6
    $\begingroup$ As the common architectures of neural networks do not have any path that actually multiplies features by each other, they instead try to get by using weighted sums and non-linear activations. This may be feasible for classification tasks, but for regression it's not. Just try training a NN to calculate $x*y$ over a wide range of x and y. Or a simpler task, learn $x*x$ (again, over a wide range of x, this is important). Unless you do log transform or other feature engineering, this is hopeless. $\endgroup$ Jan 30, 2020 at 19:27
  • 1
    $\begingroup$ @AndrisBirkmanis I agree with your comment. Seems like to do these types of interactions, you would need to design a layer that tests all interactions. For example, a CNN does not spontaneously invent convolutional filters from backprop over individual pixels. Instead, a CNN is designed with convolutional layers to automatically force these kinds of local interaction effects. $\endgroup$
    – krishnab
    Jan 8, 2021 at 22:48
  • 1
    $\begingroup$ Let's say your last layer has a softmax on top with two or more output neurons. What you are saying then is the input to the softmax is in log space (since a softmax is e^x / sum(e^x)). Thus somewhere between the input and final layer, you are kind of saying that your data is in some kind of log space. If that's true, then a + b is a multiplicative effect for some neurons in your network near that last layer, where you think the data is more or less representing log space. So... Maybe depending on your loss/activation, you already have a multiplicative model? $\endgroup$ Dec 8, 2022 at 23:35
2
$\begingroup$

To have interaction, model has to have a x1* X2, ( interaction term) , nn doesn't have exactly the same feature ( unless u create and pass it to training ), so it doesn't handle interaction explicitly. Although it will try to fit relationship using non linearity of activation functions... This is one of the reason why nn overfits.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.