4
$\begingroup$

My aim is to create segments based on survey data. This in it self is quite straight forward: I use PCA to extract information from the survey answers, and then utilize k-means to build segments in the space spanned by my PCA scores.

However, it is the next step that is posing problems. I need to be able to attach the created segments to a wider population than those in the survey. That is, I need to be able to explain either the PCAs or the segments with other data (data that is shared with both the survey and population). I'm not having much success in doing either. Ok, I realize it boils down to the fact that the survey answers simply cannot be properly explained by my other data, but how can I improve my situation?

Should I feed my other data into the segmentation process? This brings less focus to my main goal of segmenting the survey, but is it my only option? Or are there algorithms that take this problem into account? Could I use a different tactic altogether? Could EM algorithms help me identify "troublesome" individuals that should be put in a separate "unclassified" segment?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Your question sounds like a concern with the projectability of the segmentation based on the full information available from the survey to external data based on the typically incomplete and poor quality information (e.g., a few demographic factors) available from large vendors such as Experian. This is an important question to answer in doing any segmentation, i.e., does it project to external data? One way to proxy this is to match the factors in your survey to the variables from the external source -- a reduced set -- and predict. The resulting misclassification error is the test $\endgroup$
    – user78229
    Commented Nov 4, 2015 at 12:01
  • $\begingroup$ @DJohnson Thank you for your reply, could you please elaborate a bit more on the suggested test? $\endgroup$
    – Figaro
    Commented Nov 4, 2015 at 12:43
  • 1
    $\begingroup$ Ok...I'm assuming that the only information you have available to you is from the survey but that, at a future date, you will want to project the results from your segmentation to new, external data based on a reduced set of variables. What I'm suggesting is that you proxy the fit and (mis)classification rate for this new, external and unknown data based on the information you have now that matches the information you will have at some future date. Make sense? $\endgroup$
    – user78229
    Commented Nov 4, 2015 at 12:48
  • 1
    $\begingroup$ An alternative to the approach suggested by @Djohnson would be to model the survey responses (or PC scores) on predictors common to the population & the survey sample, & then perform cluster analysis of the population on their predicted responses. It's swings & roundabouts of course - you wouldn't have a misclassification problem but the clusters would be less distinct. $\endgroup$
    – Scortchi ♦
    Commented Nov 9, 2015 at 17:34

2 Answers 2

Reset to default
2
$\begingroup$

Cluster finding algorithms like k-means use a distance function to find clusters. Since you have an idea about the kind of features you want, similar to the population, use a distance function that increases the weight of the desired features and reduces weight for non essential features. The simplest distance function to start with is to set the weight for desired features to 1 and the rest to 0. This will give segments that are easy to interpret. And of course, this should be run on raw data and not dimension reduced data.

Improve this answer
$\endgroup$
0
$\begingroup$

I assume you mean things like likelihood to purchase, percentage of respondent above 18 year age, percentage of women in segment etc. by your line

I need to be able to explain either the PCAs or the segments with other data

If this is the case, lets say you have 500 responses from survey and you get 2 clusters from survey. Further if cluster 1 has a size of 200 respondents(40%) and cluster 2 has 300 respondents(60%).

Lets say in cluster 1, likelihood to purchase is 15%

Now i believe you have calculated "Error Margin" by using sample size, population size at 95 percent confidence level. Lets imagine your Error Margin is 1.96

If we repeat this exercise 100 times over with random respondents, 95 times likelihood to purchase with people who might fall into cluster 1 with similar behavioral characteristics (gender, age level or income level proportion), will be in the range of 13.04% and 16.96%

Improve this answer
$\endgroup$
1
  • $\begingroup$ Don't see how this answers the question - how to create segments such that you can fairly reliably put people into a segment without knowing all the variables that determine segment membership. $\endgroup$
    – Scortchi ♦
    Commented Nov 9, 2015 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.