I'm new to machine learning so I apologize in advance if the answer is obvious.

I have a dataset that includes an address feature. I was hoping to use one hot encoding to create a feature for each street that appears in the dataset. So if my dataset looked like:

Miles| Address
4    | 111 First St
2    | 456 Grenada Ave
9    | 789 Grenada Ave

It would become:

Miles| Address          | First_St | Grenada_Ave
4    | 111 First St     | 1        | 0
2    | 456 Grenada Ave  | 0        | 1
9    | 789 Grenada Ave  | 0        | 1

Doing this blows up the number of features out to over 1500 and puts both the train and test files from around 90 mb to over 5gb.

I was hoping to apply a RandomForest to these new datasets, but when I call forest.fit (using Python/sklearn) the memory usage balloons out to 60gb and then the Python process dies.

Couple of questions:

Is it considered bad practice to create this many features? If not - how does one go about modeling with data this size?


One hot encoding is probably the most generic approach in such a situation. However, note that a lot of features will be zero. If you have say, 1000 different streets, 999 features will be 0.

To handle this kind of situations, you should probably use a sparse representation of your matrix, if you don't do it already.


class sklearn.preprocessing.OneHotEncoder(n_values='auto',
categorical_features='all', dtype=<type 'float'>, 
sparse=True, handle_unknown='error')

You can have a sparse representation of your data, after one hot encoding. And making sure that:


Handles sparse representations (which is the case, more informations here http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)

Note that, when doing one hot encoding, it may be the case that a lot of columns are 0 most of the time (if a street appears only once or twice). It is usually better (in terms of performance of prediction of the model) to drop these scarce observations.

One hot encoding may not be the only solution in your situation. Are you trying to predict something (say, the prices of houses for the sake of the example) according to the street, or according to the location? You could also turn the adresses into their GPS coordinates (might require some efforts though) and use them as continuous predictors.

  • $\begingroup$ Wow so much good info here. I actually have the lat/long already so that is an option. I will look into sparse representations and continuous predictors. Do you know of any good resources for learning about these concepts? $\endgroup$ Sep 25 '15 at 15:27
  • $\begingroup$ Random forest will handle continuous variables for you and produce expressions such as if(lat>x1) and (lat<x2) and (lon>y1) and (lon<y2) then predict 1. With such an expression, your model is averaging over a rectangle on your map. As for sparse data, I advice you to stick to sklearn : scikit-learn.org/stable/modules/generated/… and scikit-learn.org/stable/modules/generated/… as importing various libraries in python can lead to confusion of types... $\endgroup$
    – RUser4512
    Sep 25 '15 at 15:32
  • $\begingroup$ And en.wikipedia.org/wiki/Sparse_matrix wikipedia is always a good starting point ;) $\endgroup$
    – RUser4512
    Sep 25 '15 at 15:33

Agree with @RUser4512 - is the street really the best representation of that feature? Would something else like zip code, census tract, etc. be better for the problem you're trying to solve?

If street is the appropriate representation, then SVM could be a good approach. It works well with sparse data, and is the approach Google takes with their Sofia package.

  • $\begingroup$ It may not be - I'm pretty new to all of this so, at this point, i'm trying a few different things just to see how it affects the accuracy of my model. I think zip code and census tract would not be precise enough for what I'm trying to do. I'm working at the city level so it needs to be fairly precise. $\endgroup$ Sep 25 '15 at 15:33
  • $\begingroup$ In that case, you could look into creating a geospatial grid, which is an approach taken in predictive policing. I would think that would be a better approach, since that would group together more geographically near features/events as opposed to streets, which may span the entire width/length of a city. $\endgroup$
    – Tchotchke
    Sep 25 '15 at 16:36
  • $\begingroup$ That's an interesting idea. Can you recommend any resources that would help a newbie figure out how to do that? $\endgroup$ Sep 25 '15 at 16:41
  • $\begingroup$ RAND's Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations goes over the grid mapping approach (pg. 20). I know R has a set of packages that can allow you to create the grids (it's a little involved, but doable). It takes some time to get set up with this approach, but it has been effective for predictive policing and other geospatial predictions as well. $\endgroup$
    – Tchotchke
    Sep 25 '15 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.