All Questions
14 questions
11
votes
1
answer
26k
views
K-means: How many iterations in practical situations?
I don't have industry experience in data mining or big data so would love to hear you sharing some experience.
Do people actually run k-means, PAM, CLARA, etc. on a really big dataset? Or they just ...
- 155
6
votes
1
answer
2k
views
Clustering with large number of clusters
I would like to cluster tens of millions of vectors (hidden states of BERT) into something like 20k clusters. Is there a clustering method that can do this in a reasonable time?
Standard K-means ...
- 3,469
3
votes
2
answers
3k
views
Hybrid (K-means + Hierarchical ) clustering
I have a huge dataset (50,000 2000-dimensional sparse feature vectors). I want to cluster them in to k (unknown)clusters. As hierarchical clustering is very expensive in terms of time complexity (...
- 209
2
votes
2
answers
2k
views
Silhouette clustering index in practice
I don't have much experience with data analysis algorithms (data mining, machine learning, if you like) and I'm interested if some could share their experience with practical usage of Silhouette in ...
- 2,877
2
votes
2
answers
1k
views
Evaluation of k-means output for >3D
I'm implementing the k-means algorithm (in R Map-Reduce) and I wanted to verify if the output I'm getting is close enough to the true centroids of the cluster. This is how I'm verifying with a 2D ...
- 589
2
votes
2
answers
179
views
How to find the number of clusters when more than one datasets are aggregated as one?
Suppose 3 datasets has 3 ,7, 4 clusters in their respective dataset.
When I aggregated them as one dataset what's the safest number of cluster to choose as perimeter for kmeans or any supervised ...
- 23
2
votes
0
answers
2k
views
K-Means Clustering using modified correlation (1 - pearson correlation coefficient)
I am trying to implement k-means clustering on a 6x6 data set that looks like this:
2 3 6 0 1 7
4 9 9 6 2 2
0 1 7 9 5 0
2 3 2 7 8 3
8 2 9 2 3 1
8 0 0 1 7 9
Using ...
- 21
1
vote
2
answers
8k
views
Cluster Analysis for large data in R
I am trying to perform a clustering analysis for a csv file with 50k+ rows, 10 columns. I tried k-mean, hierarchical and model based clustering methods. Only k-mean works because of the large data set....
- 21
1
vote
1
answer
1k
views
Threshold for kmeans anomaly detection
I'm learning the kmeans to find out anomaly from the dataset.
but I don't know how to set threshold.
I tried by the putting mean of the centroid to point distance but it's not working, half my record ...
- 141
1
vote
2
answers
776
views
Streaming k-means
I want to perform something like streaming/online/out-of-core kmeans clustering on large data.
Here is simple idea:
Break all data into N chunks.
Read from disk 1st chunk and calculate centroids ...
- 2,227
1
vote
1
answer
777
views
Alternative to spherical K-Means for clustering large high dimensional dataset
What are some alternatives to Spherical K-Means for clustering very large datasets of high dimension?
I'm looking for something that will be fast even on large datasets, and preferably will not ...
- 113
1
vote
1
answer
172
views
Clustering large 3D dataset into many clusters
I have a 3D point cloud with several million points and I need to partition it into roughly 50k clusters. As the clusters have to be spherical, usually a drawback of k-means, k-means seems pretty ...
- 111
1
vote
0
answers
82
views
Clustering large yearly, (presence/absence) dataframe
I have a data frame of 500,000x23 dimensions. The data is binary, representing presence or absence. The data follows identified trees through time (23 years) and looks at whether the tree is present ...
- 11
1
vote
1
answer
35
views
What is a good approach for Scalable k-means? [closed]
how can we use a distributed system (cluster of machines) to get final k-means centroids that are exactly equal to the centroids we would get if we were to process the same huge dataset on a single ...
- 19