4
$\begingroup$

I'm currently trying to learn a bit about neural nets (did Andrew Ng's Coursera course) but have a question I haven't been able to find a good "rule of thumb" answer to.

Lets say I have the classic data set of house prices, and for input data I have a bunch of data like floor plan area, number of bedrooms, number of bathrooms, age of the house.

Now, it makes sense that floor plan area would be directly correlated to predicted price (more square metres, higher price). But it is theoretically possible that the age of the house does not have a linear relationship. Instead, new build properties have a higher price because they're new and shiny, very old properties have a higher price because they're "classic" or "have character", but properties in between have a lower price (this is entirely hypothetical, just to illustrate my point). In this scenario, I'm guessing treating age as a continuous value isn't going to be able to model this (or can it?), and instead I should come up with three new buckets, "is new", "is old", "is middle" or whatever and set the appropriate bucket to 1 and the others to 0. But this would mean I'm applying my own biased notion of what constitutes new or old, and the neural net could miss out on some other relationship that I haven't spotted to do with age.

My question is, how do you know when you should create these buckets instead of using the value as continuous, and if creating buckets, how do you know how many buckets to create and which rows should go into each bucket? Or is it just trial and error, and you have to train the net with and without buckets, and with buckets of varying sizes and limits until you find the best fit?

Apologies for the long question, hopefully it makes sense!

$\endgroup$
  • $\begingroup$ A standard way would be to use splines as input. You fix the knot positions and then linear regression /nn weights fix shape of spline (it's generalisation of feeding powers of X to create polynomial regression with linear regression) $\endgroup$ – seanv507 Jul 25 '17 at 11:21
  • $\begingroup$ You might also think interactions are important eg square area x average price per meter.., age x location etc $\endgroup$ – seanv507 Jul 25 '17 at 11:23
  • $\begingroup$ Try a few synthetic experiments too. Eg how well /fast can you train z=x *y or piecewise constant function etc using nn only $\endgroup$ – seanv507 Jul 25 '17 at 11:25
  • 1
    $\begingroup$ aren't neural nets able to capture non-linear relationships? $\endgroup$ – Antoine Aug 1 '17 at 9:40
3
$\begingroup$

The non-linearity you are concerned about can be effectively handled by neural nets. That is one of the key points with using them instead of a linear model. A neural net can , at least theoretically, approximate any continuous function. It is called the Universal approximation theorem. Of course it might still be hard to learn but in practice it generally works quite well even if you don't find the optimal solution.

So, in short. No, you do not need to split the features into buckets.

Example

I'll show the non-linear problem by example.

Here is a linear dataset with two continuous features and one continuous output (the color of the dots). Hence a regression problem similar to the housing price example but more obvious.

I've trained a linear model on the data which shows as the shade behind it.

This obviously works really well in the linear case (left). But if we try to train a linear model on a non linear dataset the output doesn't look as good (right). enter image description here enter image description here

As you would expect it is not possible for the model to capture the relationship in the data. This is where you could resort to binning the data into buckets. Effectively discretizing the predictions into squares in the input space. Or if you want more continuity you can use splines for these but looking at this set you might expect that this can be quite tricky as the pattern is dependent on both features. You can easily imagine more complex structures in more high dimensional problems.

Another approach to solve the non-linearity is to add some hidden neurons to your linear model making it a neural net. Adding 3 hidden neurons will give you the following non-linear output (left) and adding another layer and a few more neurons gives you an even more accurate solution (right).

enter image description here enter image description here

The example images are generated here: http://playground.tensorflow.org. It's a great place to play around with neural nets and see what effects different parameters will have on the training and result.

$\endgroup$
  • $\begingroup$ That playground link is amazing, thanks! Interestingly, I've read that "with enough neurons a single hidden layer should be able to model any function", yet I can see from the playground that with the spiral data (since it's limited to only 8 nodes per layer) that it cannot model it with a single layer but can with 2 hidden layers. Definitely gives me something to think about in my model. In your experience, is it better to have, for example, 1 layer with 1000 nodes, or 2 layers each with 500 nodes for modelling complex data? $\endgroup$ – Matt Aug 2 '17 at 13:44
  • $\begingroup$ There is no general answer to how you should design a network as it depends on the problem. However think of the number of parameters in your example. A 1000 layer single node network will have n_in*1000*n_out parameters to learn whereas the 2 layer example will have n_in*500*500*n_out parameters which is a lot more than the single layer one. More parameters will in general have more "capacity" to learn complex patterns. But a more complex model will not always give you better result as it is harder to train and have a higher risk of overfitting. $\endgroup$ – while Aug 3 '17 at 14:30
  • $\begingroup$ But deeper structures will in general be able to model more complex relationships since you have more nonlinear steps than in a shallower structure. If you look into Convolutional deep neural nets they generally have a lot of layers that each extract different sets of features and use the features in the previous layer to find more complex features. Eg to classify between cats and dogs the first layer extract different types of lines, the next layers combine lines into angles, the third angles into ears and eyes the last ears and eyes into cats or dogs. Somewhat simplified. $\endgroup$ – while Aug 3 '17 at 14:40
1
$\begingroup$

There are probably no principled ways to determine when to create buckets or use the value as continuous like the 'age' feature, since the predictiveness of age in different tasks vary a lot.

Trial and error is always good if having enough time and computation resources. If not, manually decide how many buckets to create or how many ways of bucket creation to experiment based on intuition is usually good-performing if confident that the bucket creation well reflects experience and knowledge regarding this feature.

$\endgroup$
  • $\begingroup$ Thanks for your input. So in my case in this hypothetical scenario, if I know absolutely nothing about house prices or the age of houses, is there no algorithm/process similar to PCA that would help me decide what to do? $\endgroup$ – Matt Jul 25 '17 at 10:54
  • $\begingroup$ Sorry Tom, could you clarify your last comment? I didn't quite understand $\endgroup$ – Matt Jul 25 '17 at 11:07
  • $\begingroup$ Sorry I accidentally added comment and cannot delete. PCA always helps by creating many buckets and then use PCA to reduce dimensions. However, this slows computation and may not perform as well compared to certain ways to bucket creation depending on tasks. $\endgroup$ – Tom Jul 25 '17 at 11:09
  • $\begingroup$ The predictiveness of age, size, etc., is totally different depending on tasks. But for tasks like image processing, natural language processing, etc., intuitively there some generalizable features in different tasks, and researchers design technologies to learn more universal features in unsupervised approaches; these features can be initial features in different tasks. $\endgroup$ – Tom Jul 25 '17 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.