I have 6 months of sales data (about 12 million rows non labeled) that i need to cluster. I am going to use 4 numerical and 1 categorical (2 levels) variable. As you can imagine the amount of the data is really big so i was wondering what i can do to speed up the whole procedure and also if using k_prototypes is the best algorithm to use or if there is a better algorithm that can handle so big mixed type of data. I know that by getting a sample this will be faster but as my data set is huge how big the sample should be to be representative? Also as it is a sales data how can i be sure that i will get a representative sample?
1
-
$\begingroup$ For a clustering procedure able to handle huge number of cases and allowing both numeric and categorical variables, search the site for "Two-step (or TwoStep) cluster analysis" available in SPSS. Working with random subsamples with subsequent cross-validation (e.g. see pt 5) is also a way to go. $\endgroup$– ttnphnsDec 16, 2019 at 16:58
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
First you need to be clear on what you need. Often clustering is not that interesting once you've understood what it actually does...
I'd assume that you first need to prepare the data, for example aggregate it in a more interesting way. That will likely give you more attributes, and much fewer instances.
Then frequent itemsets and association rules are often much more interesting on sales data rather than clusters.
But your data just has one category variable... Just split your data then you can use k-means which will (as long as you use a good algorithm and not Lloyd's) easily scale to the entire data. But since it is k-means, a few thousand data points will be enough, larger data only yield diminishing returns.
Anyway, never scale to a big data set, until you know that your approach works. That would be wasted time and resources to compute something on the big data just to find out that it's not what you wanted in the first place. First use a sample to understand the problem and the solution then work on scaling the solution to the entire data.
answered Dec 17, 2019 at 7:07
-
$\begingroup$ Frequent items and association rules are not gonna work for me as the data are policy data. So my goal is to create customer segments to be able to focus on each of them separately. So the questions is still how big my sample should be to get the first results out and see if it works before i apply it to the whole dataset. $\endgroup$ Dec 17, 2019 at 11:24
-
$\begingroup$ Split the data on the binary. Then use k-means. Runtime will be tiny anyway, and size does not help much. 10k samples should be enough for testing. $\endgroup$ Dec 18, 2019 at 0:04
-
$\begingroup$ If testing gives me good results should i rerun that on the whole dataset or should i just predict the values for the rest of the data? $\endgroup$ Dec 18, 2019 at 11:54
-
$\begingroup$ On such low dimensional data, the result will likely not change much when you go to the entire data set. But you should be able to afford it (use the centers of the sample as initial centers for the full data!). You can even afford to verify this - how much different are the results with both approaches? $\endgroup$ Dec 18, 2019 at 19:27
-
$\begingroup$ But you'll quickly see that getting verifiable and good results will be your main problem, not scaling it to the entire data. $\endgroup$ Dec 18, 2019 at 19:28