1
$\begingroup$

I have a population of ~500 000 individuals. Each of them either own or don't own a bike, and only 500 own one. I need to divide my population based on several criteria (age, home-town, own a car, etc...), but I have many criteria and I could end up with groups of only one individual.

So I have two questions:

  • How do I choose the order of criteria for the splitting : e.g., should I split first by "age" and then by "home-town" or the opposite? What tool can I use to find that?

  • When do I stop splitting, e.g. what is the minimal representative population for a group ?

I found a very large number of ways to determine a sample population, and I am a bit lost about how to choose the best one for my case. My issue is that I have only 0.1% of the population who owns a bike.

As for the splitting method, I though about ordering my criteria in ascending number of possible outcomes : home-town may have 1000 different outcomes, but "own a car" only have 2 : True or False. So I was thinking about splitting by "own a car" first, then by "age" and then by "home-town". That being said, I think there is a better (and proper) way to handle this.

EDIT :

  • Why do want to split your data?

I want to see if some groups (defined by a set of characteristics) are more susceptible to buy bikes than other groups.

  • In how many sets do you want to split the data?

As much as possible, given there is enough individual to remain representative

  • Do these datasets need to be of (roughly) equal size?

Not necessarily.

Improve this question
$\endgroup$
1
  • $\begingroup$ First some questions: Why do want to split your data? In how many sets do you want to split the data? Do these datasets need to be of (roughly) equal size? please elaborate by editing your question $\endgroup$
    – IWS
    Commented Jan 26, 2017 at 12:16

1 Answer 1

Reset to default
0
$\begingroup$

It really depends on the machine learning tactic that you would like to employ. This seems like a prototypical problem for a random forest using binary classification.

In this case, you would choose splits in the data that increases the purity the of the child groups the most. Here's a good article on this: http://people.revoledu.com/kardi/tutorial/DecisionTree/how-to-measure-impurity.htm

You continue this process iteratively until the tree is entirely built and all leaves (sub groups of sub groups) have only 1 person in it (bike owner or not). By building a number of trees, you can average their predictions and get very robust results.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.