I am working on an unsupervised learning problem and can use a little help for the same! Objective: To find out significant payment patterns or segments of customer who tend to have similar payment patterns for products which are sold on EMI with a given tenure.
Data description! I have customer level details of payments made by each customer during his lifecycle. The life cycle of a customer is defined as Unit Promotional Age (UPA) which simply days since purchase divided by the days of payment plan i.e. the number of days in which the customer is expected to pay the whole amount of the product which is considered as a categorical variable in this case. The features or variables selected for the algorithm are as follows: Average payment amount of the customer Min/max payment amounts Days since 1st transaction Days since last transaction Average days between payments Max and min days between payments Payment made at each UPA (unit promotional age) with 21 levels starting from 0 to 2+ with a stepsize of 0.1 which are transformed into dummy variables for example amount_0, amount_0.1,... Amount_2+ Total number of payments
Approach: The data has high variance. A PCA was done to reduce the dimensionality and 4 PCs explaining close to about 60% of the variance were selected for a Kmeans clustering algorithm. The dummy variables as explained earlier shows high correlation with each other. A k value of 3 is selected for clustering by looking at the elbow plot of cluster errors. The PCs are mostly weighs the amount_UPA very highly. The dataset consists of data from 3 countries also all the customer may or may not have reached 2+ UPA. The clusters formed are having %age customers in it as 65%, 30% and 5% (which I think is pretty disproportionate) the characteristics of each cluster tends to weigh the amount paid at each UPA highly with difference in average for each cluster but these have high standard deviation as well (value higher than that of mean)
Questions: 1. Should I consider multi collinearity into account while feature selection 2. How to tackle the high variance in data and in the cluster properties 3. Should I go for other algorithms which are better with data with high variance, is there any? Should EM, DBSCAN help? 4. How can I design an intervention process based on characterstics of segments such as prioritizing which customers to call and when to call