I have a population of ~500 000 individuals. Each of them either own or don't own a bike, and only 500 own one. I need to divide my population based on several criteria (age, home-town, own a car, etc...), but I have many criteria and I could end up with groups of only one individual.

So I have two questions:

  • How do I choose the order of criteria for the splitting : e.g., should I split first by "age" and then by "home-town" or the opposite? What tool can I use to find that?

  • When do I stop splitting, e.g. what is the minimal representative population for a group ?

I found a very large number of ways to determine a sample population, and I am a bit lost about how to choose the best one for my case. My issue is that I have only 0.1% of the population who owns a bike.

As for the splitting method, I though about ordering my criteria in ascending number of possible outcomes : home-town may have 1000 different outcomes, but "own a car" only have 2 : True or False. So I was thinking about splitting by "own a car" first, then by "age" and then by "home-town". That being said, I think there is a better (and proper) way to handle this.


  • Why do want to split your data?

I want to see if some groups (defined by a set of characteristics) are more susceptible to buy bikes than other groups.

  • In how many sets do you want to split the data?

As much as possible, given there is enough individual to remain representative

  • Do these datasets need to be of (roughly) equal size?

Not necessarily.

  • $\begingroup$ First some questions: Why do want to split your data? In how many sets do you want to split the data? Do these datasets need to be of (roughly) equal size? please elaborate by editing your question $\endgroup$ – IWS Jan 26 '17 at 12:16

It really depends on the machine learning tactic that you would like to employ. This seems like a prototypical problem for a random forest using binary classification.

In this case, you would choose splits in the data that increases the purity the of the child groups the most. Here's a good article on this: http://people.revoledu.com/kardi/tutorial/DecisionTree/how-to-measure-impurity.htm

You continue this process iteratively until the tree is entirely built and all leaves (sub groups of sub groups) have only 1 person in it (bike owner or not). By building a number of trees, you can average their predictions and get very robust results.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.