1
$\begingroup$

I have a dataset that has 5 variables (columns). These are median house price, median income, number of people with no educational degree, number of people with high school degree, number of people with university degree. I have these variables by different census sub-divisions. So rows represent different census sub-divisions. I am interested to segment this data into smaller number of segments e.g. (Low, Average, High) income, (Low, Average, High) house, (Highly or Less) educated.

In other words, I want to e.g. find a way to define ranges for (Low, Average, High) income levels in following two cases:

  1. By considering the geographical location i.e. census sub-divisions
  2. Without considering the geographical location i.e. census sub-divisions.

At first I thought that maybe k-means clustering is appropriate (at least for the 2nd case above where I am not considering the census sub-divisions). But this does not seem right to me as I am having both prices/incomes and number of people in the dataset. Also this approach gives me centers of each cluster and not actually the ranges for each levels: Low, Average, and High.

Is there any other approach I can use?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Would it be fair to say that you are interested in a supervised learning algorithm where income is the target and the other features are predictors of income? Next, you want to do it two ways: 1) based on a model that includes geography in @anony-mousse's sense (below) in terms of proximity or nearest neighbor and, 2) a model that ignores geography, correct? $\endgroup$
    – user78229
    Commented Apr 7, 2016 at 19:44
  • 1
    $\begingroup$ Not exactly, I don't have any response variable here. So the answer to your 1st question is no. I am interested in two cases: with and without considering the geographic correlation among variables. $\endgroup$
    – Stat
    Commented Apr 8, 2016 at 1:39
  • $\begingroup$ Ok...but how do you propose integrating geography in a "correlation?" $\endgroup$
    – user78229
    Commented Apr 8, 2016 at 1:51

1 Answer 1

Reset to default
1
$\begingroup$

There are plenty of approaches.

Instead of relying on k-means that is sensitive to outliers, why don't you just consider values below the 1QR to be low (i.e. the botrom 25%) and values larger than the 3QR (i.e. the top 25%) to be high? IQRs are rather robust statistics. You cold also use low=below $\mu-\sigma$ and high=above $\mu+\sigma$ where $\mu$ is the mean and $\sigma$ is the standard deviation. But this won't work well on skewed data (and income probably is skewed).

For geography, you can do this with respect to the neighbors only.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I have already thought of this approach. But this ignores the correlations among variables which is not good I guess. $\endgroup$
    – Stat
    Commented Apr 8, 2016 at 1:43
  • 1
    $\begingroup$ So does k-means. It optimizes the sum of variances, but it does not pay any spwcial attention to correlation. And if you want low/med/high in ever attribute, you would need to cluster them separately. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Apr 8, 2016 at 6:05
  • $\begingroup$ Maybe I didnt use the term "correlation" properly here. What I meant is that e.g. in the k-means we are finding the distances in a multivariate sense. So geographical coordinates in combination of income, education, etc. $\endgroup$
    – Stat
    Commented Apr 9, 2016 at 14:34
  • $\begingroup$ k-means only works with squared Euclidean, not on geographic distance like haversine and won't produce "levels". $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Apr 9, 2016 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.