0
$\begingroup$

I'm trying to do customer segmentation by using PCA to reduce dimensionality and then feeding the resulting principal components into a K-means algo to get at the final segments. Some of my variables are percentages. For example, conditional on arrive on a particular section of the site, what percentage when on to purchase something. The problem with these percentage variables is that they are not define for people who never navigated to that section of the site; so for those people, I forced their percentage to be 0.

pr(purchase|landing on section A) = ifelse(#times landing on section A > 0, 
                                           #purchases/#times landing on section A*100, 
                                           0)

Is this the right way to treat a percentage variable in clustering/unsupervised learning if some values are undefined? One problem in forcing the undefined value to be 0 is that I'm basically telling the algorithm that these people are the same as those who landed on section A but did not make any purchases.

Another solution that I thought is to include the complement of the above probability so that the people who did not land on section A are identified by when these two percentages are both 0, i.e. include both

pr(purchase|landing on section A) = ifelse(#times landing on section A > 0, 
                                           #purchases/#times landing on section A*100, 
                                           0)

and

pr(did not purchase|landing on section A) = ifelse(#times landing on section A > 0, 
                                           100-pr(purchase|landing on section A), 
                                           0)

Note that if everyone landed on section A then I would only need to include one of them because then pr(purchase|landing on section A) = 100 - pr(did not purchase|landing on section A); but this relationship doesn't hold if I'm forcing these percentages to take on 0 when they are not defined (for customers who never landed on section A).

$\endgroup$
0
$\begingroup$

The 2nd case presented would also have problem when a user did purchases everytime he landed on the section. That is 100% buy rate. Although from a practical purpose, we may consider assuming that the chances that a user will do a purchase every time he lands on a section is very unlikely, in that case your 2nd formulation may work.

Options

  1. If there are no data points with 100% buy rate then you can probably use the 2nd formulation.
  2. If there are some data points with 100% buy rate then I think it would be good to keep the 2 features separate, scale them and then use them in your PCA/Kmeans implementation.
$\endgroup$
  • $\begingroup$ Sinha my concern isn't so much the scenario of 100% buy rate as much as customers who never landed on section A and therefore their probabilities are undefined. Right now I'm forcing these undefined probabilities to be 0, but then that's telling the algorithm that those with 0% buy rate is the same as those who never landed on section A. If everyone has landed on section A then using either one is fine because the pr(purchase|landing on section A) = 100 - pr(did not purchase|landing on section A) $\endgroup$ – Amazonian Apr 11 at 3:00
0
$\begingroup$

The use of PCA creates more problems than it's worth here.

I'd also not force probabilities to a 0-100 scale, but keep them 0-1.

NA values logically make all variables NA after PCA, because it creates some dependency of all variables on the undefined value. See for example the discussions, comments, and links here: Imputation of missing values for PCA

For k-means, dimensionality does not make that much of a difference. In spherical k-means it is common to use k-means with some 50000 variables! So you should be able to just use k-means here. Supporting NA values in k-means is much cleaner. You only need to make sure your initial centers don't have NAs. One can argue that it's okay to omit it when finding the closest center. And when recomputing the centers, one can omit the NAs, and keep the old coordinate when all points were NA.

Plus, the results will remain more interpretable if you don't apply PCA first.

$\endgroup$
  • $\begingroup$ can you provide more details or a reference on how to handle NAs in k-means? $\endgroup$ – Amazonian Apr 11 at 7:22
  • $\begingroup$ No, I don't have a reference or implementation. Bit that should be straightforward to do. $\endgroup$ – Anony-Mousse Apr 11 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.