Which clustering algorithm to use for dataset with high variance

I am working on an unsupervised learning problem and can use a little help for the same! Objective: To find out significant payment patterns or segments of customer who tend to have similar payment patterns for products which are sold on EMI with a given tenure.

Data description! I have customer level details of payments made by each customer during his lifecycle. The life cycle of a customer is defined as Unit Promotional Age (UPA) which simply days since purchase divided by the days of payment plan i.e. the number of days in which the customer is expected to pay the whole amount of the product which is considered as a categorical variable in this case. The features or variables selected for the algorithm are as follows: Average payment amount of the customer Min/max payment amounts Days since 1st transaction Days since last transaction Average days between payments Max and min days between payments Payment made at each UPA (unit promotional age) with 21 levels starting from 0 to 2+ with a stepsize of 0.1 which are transformed into dummy variables for example amount_0, amount_0.1,... Amount_2+ Total number of payments

Approach: The data has high variance. A PCA was done to reduce the dimensionality and 4 PCs explaining close to about 60% of the variance were selected for a Kmeans clustering algorithm. The dummy variables as explained earlier shows high correlation with each other. A k value of 3 is selected for clustering by looking at the elbow plot of cluster errors. The PCs are mostly weighs the amount_UPA very highly. The dataset consists of data from 3 countries also all the customer may or may not have reached 2+ UPA. The clusters formed are having %age customers in it as 65%, 30% and 5% (which I think is pretty disproportionate) the characteristics of each cluster tends to weigh the amount paid at each UPA highly with difference in average for each cluster but these have high standard deviation as well (value higher than that of mean)

Questions: 1. Should I consider multi collinearity into account while feature selection 2. How to tackle the high variance in data and in the cluster properties 3. Should I go for other algorithms which are better with data with high variance, is there any? Should EM, DBSCAN help? 4. How can I design an intervention process based on characterstics of segments such as prioritizing which customers to call and when to call

• Suggestion: Plot the raw data first, then plot the transformed data (PCA, power/log/arcsin transforms, whatever) and then think about algorithms to cluster the data. – usεr11852 Jul 19 '18 at 20:18
• Had plotted the raw data, can't really find any pattern. The scatter plot is really cluttered and clumsy with .3 million data points, the P CA is still less cluttered and the plot between PCA 1 and 2 with cluster tagging are distinct but not uniquely separated. – Arindam Bose Jul 19 '18 at 20:32

• When you are doing PCA, try to retain ~90% information rather than 60%
• Check the data prep part. Remove outliers, as k-means can't really handle them well also check try different normalization and transforms (log)
• Keep only one variable if number of variables are highly correlated (>0.7)
• The clusters are formed based on data patterns, so check even if the payments have high variance within a cluster but some other variable might be clearly separated.
• End of the day clusters are formed based on data actual data pattern which may not be what you hope for (hence the name unsupervised)
• 1. If I consider taking 90% of the variables, I end up taking almost all the PCAs except for maybe 1 or 2. As mentioned earlier, the data has high variance, I tried removing outliers with the method abs(x-x.mean)>3*x.std for all the variables and i ended up excluding 20% of my population. Hence to keep the excluded population in a limit of 7% I had taken 5*x.std. My correlation matrix has a range of .3 to -.3 should I still remove correlated variables? I will try normalization techniques though. – Arindam Bose Jul 20 '18 at 4:04
• 1. If I consider taking 90% of the variables, I end up taking almost all the PCAs except for maybe 1 or 2. As mentioned earlier, the data has high variance, I tried removing outliers with the method abs(x-x.mean)>3*x.std for all the variables and i ended up excluding 20% of my population. Hence to keep the excluded population in a limit of 7% I had taken 5*x.std. My correlation matrix has a range of .3 to -.3 should I still remove correlated variables? I will try normalization techniques though. – Arindam Bose Jul 20 '18 at 4:05
• For PCA, better to keep more but that's okay. The SD is not a great idea for long-tailed distributions, try using percentile (take out below 5th and after 95th based on distribution). Correlation is low so no need to do anything. – Nishad Jul 20 '18 at 5:41

Don't just stack functions to help to eventually get a random result. There is no reason to assume that clusters should have similar size!

Instead, begin with the property that you want to achieve. What is a good clustering in your domain? You need to get this into a function to evaluate your clustering. Until you can evaluate the results in a way specific for your problem, you won't get anywhere. You'll just be doing random things and not solving anything.

Once you have formalized your objective, choose your tools such as PCA or k-means based on whether they actually can improve this measure. Because the chances are they don't. And then don't use them.

I.e. put your problem first, not the tools.