based on customer data I want to perform a clustering using different clustering algorithms (K-Means, Expectation Maximization, etc.) in R. The most attributes were engineered pursuing the goal to be basically meaningful for a customer segmentation.

Without feature selection, the results are very poor regarding evaluation criteria like ASW, BSS, WSS, etc.

My question now is whether I need to do a feature selection technique (wrapper/filter) or just select the features I think are most valuable for segmenting the customers. I found very different sources regarding this issue. The most authors say the features have to be selected concerning the business objective. Other sources propose feature selection methods for unsupervised learning. Is that really useful for a customer segmentation or is it only needed for image segmentation for instance?

My opinion is: Attributes might be economic valuable even so not useful for the clustering process and vice versa. This would mean I select manually the features.

I performed already a PCA which resulted also in poor results regarding clustering evaluation criteria. Therefore I obviously have to select only a few attributes in order to obtain a clear and stable clustering.

migrated from stackoverflow.com Jan 16 '16 at 7:49

This question came from our site for professional and enthusiast programmers.

While @mtoto is right and this is more suited to stats.stackexchange.com, as I happen to use R for business research, I'll give you my take as answer.

Unsupervised learning is not suitable for your objective. Customer segmenting absolutely has to be related to the business goals.

One question you can ask yourself to help guide the features to choose is, when you say "based on customer data", what data were gathered and why were those data gathered rather than other data? Usually the customer data are collected in the first place with some sort of business objective in mind. The result of your consideration may also be to refine the data collection process to only collect data relevant to your clustering (unless it's a larger pool of customer data that has many intended uses).

Customer segmentation is usually based on features that will predict product/service selection and quantity. All other features are usually less relevant to segmentation. The specifics depend on the nature of the products/services offered by your company and your customers' reactions to those products/services. If your goal is pricing strategy, i.e., determining optimal prices for different customer segments, then you'll want to focus on features that relate to the utility function of customers, i.e., the perceived value they get out of your product/service relative to the amount they pay. If your goal is product/service expansion, i.e., investigating where to add new products/services for customers, then you'll want to focus on features that relate to particular customer needs in the different segments and the pairing of your product/service to each of those needs, identifying gaps.

Once you've got your feature selection narrowed down to your business goals, start with basic linear models (or even just correlation) to validate your assumptions that those features are indeed related to the desired dependent variables. Once you've established correlation around core features, you can then add additional, plausibly-related features, and run clustering tests piecemeal to determine which other features are redundant (i.e., the same as the previously chosen core features) and which ones expand upon them in a way that increases the overall r^2 of the desired independent->dependent variable relationship. As you validate individual features in this manner, you can progressively build larger cluster models. These models are most useful for telling you when to not bother collecting/evaluating customer data that are barely incrementally different from other core factors.

At the end of the day, in my experience, there are usually very few features that relate to a particular business goal. In one project, I assessed around 46 different features, and only found 6 that mattered. That's also why unsupervised learning models aren't great--They try to hard to select too many features, and you end up with either spurious results, or results that are uninterpretable in terms of business actions to take--which is your whole purpose.

I suggest reviewing the literature / work by the mixmod group and by Raftery's group. Both have methods for model-based clustering involving both feature selection and without feature selection. Heuristic based methods may be appropriate for your but the performance of both heuristics, and the model based methods, tend to be highly influenced by your data inputs and your data pre-processing (as below).

Typically in a business case, you have variables from many different distributions. This poses problems in mixture modeling; and, you have not specified (a) if this is (or isn't) the case in your data, and (b) (if so) how you wish to deal with it. Another concern is how knowledgeable you are about your data. How confident are you that you can actually select the most important features?

Questions

  • What types of variables do you have? What are there distributions?
  • What is there correlation structure (you mentioned poor results, without detail, from PCA)?
  • How are you pre-processing your variables?

If you provide additional detail on your data, a more complete answer can be provided.

  • Thanks for your answer. The most frequent type is numeric, whereas two variables are nominal. For K-Means, these two were not considered. Pre-processing is done according to a book (e. g. removing outliers and records with missing values, etc.). also, correlations and the distributions were looked at. Variables with a higher correlation than 0.9 as well as variables with a "poor distribution" (90% or more of the records having the same value) were removed. I'm very knowledgable about my data because I extracted the most variables based on basis variables by myself. – whitenight120 Jan 14 '16 at 20:38
  • I must have been unclear... Numeric is a data type, not a distribution. Are they all normally distributed? Poisson? Is overdispersion present? Etc.... Given that your question is unspecific,I think mine is a sufficient answer and points you to some of the appropriate literature – Alex W Jan 15 '16 at 0:01
  • You asked for "What types of variables do you have". which I only answered. There is a normal distribution for all variables. I forgot another step in the preprocessing: z-standardization. Regarding literature, I pored over so many books... – whitenight120 Jan 15 '16 at 8:13

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.