For example, one wants to predict house prices and have two input features the length and width of the house. Sometimes, one also includes 'derived' polynomial input features, such as area, which is length * width.

1) What is the point of including derived features? Shouldn't a neural network learn the connection between length, width and price during the training? Why isn't the third feature, area, redundant?

In addition, sometimes I also see that people run genetic selection algorithms on the input features in order to reduce their number.

2) What is the point of reducing the input features if they all contain useful information? Shouldn't the neural network assign appropriate weights to each input feature according to its importance? What is the point of running genetic selection algorithms?


1): Including derived features is a way to inject expert knowledge into the training process, and so to accelerate it. For example, I work with physicists a lot in my research. When I'm building an optimization model, they'll give me 3 or 4 parameters, but they usually also know certain forms that are supposed to appear in the equation. For example, I might get variables $n$ and $l$, but the expert knows that $n*l$ is important. By including it as a feature, I save the model the extra effort of finding out that $n*l$ is important. Granted, sometimes domain experts are wrong, but in my experience, they usually know what they're talking about.

2): There are two reasons I know of for this. First, if you have thousands of features supplied (as often happens in real world data), and are short on CPU time for training (also a common occurrence), you can use a number of different feature selection algorithms to pare down the feature space in advance. The principled approaches to this often use information-theoretic measures to select the features with the highest predictive power. Second, even if you can afford to train on all the data and all the features you have, neural networks are often criticized for being 'black box' models. Reducing the feature space in advance can help to mitigate this issue. For example, a user looking at the NN cannot easily tell whether a weight of 0.01 means "0, but the optimization process didn't quite get there" or "This feature is important, but has to be reduced in value prior to use". Using feature selection in advance to remove useless features makes this less of an issue.


1) Most neural networks cannot perform multiplications; they can only calculate sums (which are then individually fed through through an activation function). They must instead estimate those multiplications if they are important, which requires a lot of neurons, especially if the factors can span large ranges.

If it would turn out that the house area is in fact an important feature, you will help the network if you provide it with the area, because it can then use the neurons it would have required to estimate the multiplication of the width and the length to do other things.

Hence, including polynomial features may in some cases be beneficial to the network, but has in other cases no significant effect. Furthermore, polynomial features are only one type of derived features that may be helpful to the network. Another type of derived feature that may turn out to be helpful is for example the logarithms of the input variables (considered they are positive) which the network also must estimate to obtain.

An idea would be to allow the network to perform more operations between numbers than only additions, to enable it to efficiently calculate things like polynomial features itself, but it is not clear how that would work. One architecture that looks like it does something similar is the sum-product network.

2) Except from the computational cost which John mentioned, increasing the number of parameters in the model, which inevitable happens when you introduce more inputs, also increases the risk for the network to overfit, especially if you have little training data.

However, this can be made into much less of a problem if a good regularization method is used. (Dropout seems to work extremely well for that) Theoretically, with a good enough regularization method, overfitting shouldn't be a problem at all. As Hinton points out, a human has in the order of 10^14 synapses in the brain (corresponding to the connections in the neural network), but only lives in the order of 10^9 seconds, but we still seem to be able to generalize quite well. So clearly, having many parameters that can be tuned should with the right algorithm only be an advantage.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.