0
$\begingroup$

I have a data set where the samples are people and the feature are their age, sex, location, job, height, weight… Then I will have a new person with the same information and my goal is the find the twenty closest persons from him/her.

I think a clustering algorithm on Python could do the job. But there is a lot of different clustering algorithms and I don’t know which one will fit my problem the best.

KMeans seems to be the easiest clustering method but I don’t know the number of clusters. Same problem with Ward hierarchical clustering.

Mean shift seems more advanced and complicated, I haven’t found a lot of documentation on it. Then my data set is about $100 000$ samples and the maximum recommended is $10 000$ for MeanShift but I could easily separate the data set per area to reduce its size.

Do you think a clustering method is a good idea ? Which one should be the best for my problem ?

|cite|improve this question
$\endgroup$
  • $\begingroup$ there are heuristics that can be used to determine the best level of the hierarchy in hierarchical clustering (see section 3.2 of factominer.free.fr/docs/HCPC_husson_josse.pdf). I recommend the FactoMineR R package. $\endgroup$ – Antoine Jul 9 '16 at 9:19
  • $\begingroup$ You describe a similarity search, not clustering. $\endgroup$ – Anony-Mousse Jul 10 '16 at 1:30
2
$\begingroup$

If you're looking for the 20 most similar persons I would recommend a simple k-NN classifier with k = 1. That means all instances of a person are represented by only one (perhaps big) feature vector. After all distances are calculated regarding all feature vectors (= persons) you just sort them in an ascending order and pick the 20 nearest neighbours. That's it...

|cite|improve this answer
$\endgroup$
  • $\begingroup$ That's a good idea. Do you recommend a particular library to do this ? $\endgroup$ – mitsi Jul 9 '16 at 10:05
  • $\begingroup$ You don't need a library for a simple k-NN classifier. You can implement it in less than 8 lines of code. The core of a k-NN is distance function. I recommend to use something simple like a Manhatten distance function: $/sum_{i=0}^n |x_{i} - y_{i}|$ $\endgroup$ – Unhandled exception Jul 9 '16 at 10:39
  • $\begingroup$ This is a good idea but I don't think it's quite as simple here, because the data includes continuous and categorical data. I think that some effort will need to be put into defining the distance function. $\endgroup$ – Peter Flom - Reinstate Monica♦ Jul 9 '16 at 13:53
  • $\begingroup$ I see....well try out one of the surveys here: bfy.tw/6fKN $\endgroup$ – Unhandled exception Jul 9 '16 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.