I have a dataset that has 5 variables (columns). These are median house price, median income, number of people with no educational degree, number of people with high school degree, number of people with university degree. I have these variables by different census sub-divisions. So rows represent different census sub-divisions. I am interested to segment this data into smaller number of segments e.g. (Low, Average, High) income, (Low, Average, High) house, (Highly or Less) educated.

In other words, I want to e.g. find a way to define ranges for (Low, Average, High) income levels in following two cases:

  1. By considering the geographical location i.e. census sub-divisions
  2. Without considering the geographical location i.e. census sub-divisions.

At first I thought that maybe k-means clustering is appropriate (at least for the 2nd case above where I am not considering the census sub-divisions). But this does not seem right to me as I am having both prices/incomes and number of people in the dataset. Also this approach gives me centers of each cluster and not actually the ranges for each levels: Low, Average, and High.

Is there any other approach I can use?

  • 1
    $\begingroup$ Would it be fair to say that you are interested in a supervised learning algorithm where income is the target and the other features are predictors of income? Next, you want to do it two ways: 1) based on a model that includes geography in @anony-mousse's sense (below) in terms of proximity or nearest neighbor and, 2) a model that ignores geography, correct? $\endgroup$ – DJohnson Apr 7 '16 at 19:44
  • 1
    $\begingroup$ Not exactly, I don't have any response variable here. So the answer to your 1st question is no. I am interested in two cases: with and without considering the geographic correlation among variables. $\endgroup$ – Stat Apr 8 '16 at 1:39
  • $\begingroup$ Ok...but how do you propose integrating geography in a "correlation?" $\endgroup$ – DJohnson Apr 8 '16 at 1:51

There are plenty of approaches.

Instead of relying on k-means that is sensitive to outliers, why don't you just consider values below the 1QR to be low (i.e. the botrom 25%) and values larger than the 3QR (i.e. the top 25%) to be high? IQRs are rather robust statistics. You cold also use low=below $\mu-\sigma$ and high=above $\mu+\sigma$ where $\mu$ is the mean and $\sigma$ is the standard deviation. But this won't work well on skewed data (and income probably is skewed).

For geography, you can do this with respect to the neighbors only.

  • $\begingroup$ I have already thought of this approach. But this ignores the correlations among variables which is not good I guess. $\endgroup$ – Stat Apr 8 '16 at 1:43
  • 1
    $\begingroup$ So does k-means. It optimizes the sum of variances, but it does not pay any spwcial attention to correlation. And if you want low/med/high in ever attribute, you would need to cluster them separately. $\endgroup$ – Anony-Mousse Apr 8 '16 at 6:05
  • $\begingroup$ Maybe I didnt use the term "correlation" properly here. What I meant is that e.g. in the k-means we are finding the distances in a multivariate sense. So geographical coordinates in combination of income, education, etc. $\endgroup$ – Stat Apr 9 '16 at 14:34
  • $\begingroup$ k-means only works with squared Euclidean, not on geographic distance like haversine and won't produce "levels". $\endgroup$ – Anony-Mousse Apr 9 '16 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.