0
votes
2answers
36 views

Streaming k-means

I want to perform something like streaming/online/out-of-core kmeans clustering on large data. Here is simple idea: Break all data into N chunks. Read from disk 1st chunk and calculate centroids ...
1
vote
1answer
35 views

Alternative to spherical K-Means for clustering large high dimensional dataset

What are some alternatives to Spherical K-Means for clustering very large datasets of high dimension? I'm looking for something that will be fast even on large datasets, and preferably will not ...
0
votes
0answers
22 views

Clustering algorithm for my situation?

Here is my situation. I have a corpus of over 500,000 news. Now I need to cluster the news based on closeness in time and cosine similarity, using vector-space model and TF-IDF weights. I want to ...
2
votes
2answers
107 views

How to summarize and understand the reults of DBSCAN clustering on big data?

Many clustering algorithms can be used with big data, eg. versions of KMeans, DBSCAN based on Hadoop, etc. But, with k means we will get k centroids for k clusters and we can map them to the space and ...
1
vote
1answer
117 views

What data structure to use for my cluster analysis or what cluster analysis to use for my data?

I have a large dataset of categorical variables. The data consists of shoppers who purchased two items during a single trip to a store. There are approximately 75,000 cases and 1,500 different ...
0
votes
0answers
30 views

Can I use the variance of a set of observation as heuristic to decide how many times repeat an experiment?

I am applying a clustering algorithm (K-means) to a huge set of high dimensional data points (SIFT descriptors). The algorithm is not deterministic and its results depend on the initialization of the ...
2
votes
1answer
102 views

Cluster many thousands observations (mixed variable types). Cluster subsample and then classify the rest observations?

I'm trying to run a cluster analysis on a large dataset (70k+ observations to cluster) with mixed variables (numeric, ordinal, binary and nominal). I don't think I can create the distance matrix using ...
4
votes
0answers
67 views

Which variables are driving correlations within groups

I'm running an analysis on a few data sets that each typically have 100-200 cases measured across 120-160 variables - something similar to looking at gene expressions. Each variable is a non-centered ...
4
votes
2answers
876 views

Cluster Big Data in R and Is Sampling Relevant?

I'm new to data science and have a problem finding clusters in a data set with 200,000 rows and 50 columns in R. Since the data have both numeric and nominal variables, methods like K-means which ...
4
votes
0answers
100 views

Cluster on high dimensional categorical data (Images with keywords)

We're looking for clues to perform a Cluster Analysis in a DB with +400K images which have keywords associated to them. Each image could have from 1 to 30 keywords. Total keywords count is +35K. ...
1
vote
2answers
172 views

Evaluation of k-means output for >3D

I'm implementing the k-means algorithm (in R Map-Reduce) and I wanted to verify if the output I'm getting is close enough to the true centroids of the cluster. This is how I'm verifying with a 2D ...
1
vote
2answers
322 views

Clustering large scale data in fine-grained clusters

I've got large amount of data (e.g. 100K) and I want to cluster them in very fine-grained clusters (e.g. 10K). I look for an appropriate algorithm that uses the similarity function instead of whole ...
2
votes
0answers
184 views

Clusters produced by R intersect

I am new here - and relatively new to statistics, data mining and R. I am trying to understand why my data is not clustering correctly - or if I am reading it wrong. Shortly about the project: My ...
1
vote
2answers
489 views

Hybrid (K-means + Hierarchical ) clustering

I have a huge dataset (50,000 2000-dimensional sparse feature vectors). I want to cluster them in to k (unknown)clusters. As hierarchical clustering is very expensive in terms of time complexity ...
4
votes
1answer
436 views

Suggestions for multi-dimensional clustering

I am working in a genomics project and I ended up having a huge table with around 800 measurements (cases/rows), around 200 channels (columns/continuous variables) and 5 categories (one categorical ...
3
votes
2answers
408 views

Clustering of 10's of millions of high dimensional data

I have a set of 50 million text snippets and I would like to create some clusters out of them. The dimensionality might be somewhere between 60k-100k. The average text snippet length would be 16 ...
2
votes
1answer
138 views

What is heavy hitter analysis?

I have some data on the number of times each of my machines turned off (due to an error) in a particular time period. There are about 6 different classes of machines being used to construct a total ...
4
votes
3answers
369 views

Looking for sparse and high-dimensional clustering implementation

I'm looking for a clustering implementation with the following features: Support for high-dimensional data. Now I have approximately 160.000 dimensions/features. Be able to manage sparse matrix. ...
9
votes
3answers
167 views

Space-efficient clustering

Most clustering algorithms I've seen start with creating a each-to-each distances among all points, which becomes problematic on larger datasets. Is there one that doesn't do it? Or does it in some ...
7
votes
4answers
308 views

Clustering of large, heavy-tailed dataset

I have a dataset of 130k internet users characterized by 4 variables describing users' number of sessions, locations visited, avg data download and session time aggregated from four months of ...