Questions tagged [k-means]
k-means is a method to partition data into clusters by finding a specified number of means, k, s.t. when data are assigned to clusters w/ the nearest mean, the w/i cluster sum of squares is minimized
272 questions with no upvoted or accepted answers
2
votes
0
answers
155
views
Anomaly Analysis (K-Means) - finding suspicious activities/operators
I am relativly new to the field of data mining and want to make a anomaly detection on transactional retail data. I want to use a simple anomaly detection (kmeans at the moment) for finding suspicious ...
- 93
2
votes
0
answers
245
views
Vector Quantization of heavy tailed distribution
I'm generating with Monte Carlo simulation some stock price $X$.
Once I have the stock price sample, I want to cluster it with 100 points $\hat{X}$.
My problem is that the error associate with my k-...
user21186
2
votes
1
answer
99
views
What's the methodology behind the most-difference-between-groups-tag-cloud?
What is the likely stats methodology used in this old OKCupid post?:
http://www.economist.com/blogs/johnson/2010/10/sexuality_and_language
And this:
http://blog.okcupid.com/index.php/the-real-stuff-...
- 543
2
votes
0
answers
457
views
Standardizing non-normal data for use in distance-based classifier
I have a dataset containing non-normally distributed variables that I want to feed into a distance-based classifier (e.g. K-means). Is it ok to just subtract the mean and divide by the standard ...
- 155
2
votes
0
answers
1k
views
Using linear discriminant analysis to validate the cluster groups resulting from kmeans
I'm currently working on a cluster analysis project and ran kmeans on the data for k=2.
I was reading similar articles on similar experiments, and the investigators used discriminant analysis to ...
- 873
2
votes
0
answers
10k
views
Distance threshold for clustering
Usually online clustering methods (based on kmeans or not) define a distance threshold value. If a new data-point $x$ is far enough from the nearest center $c$ (i.e. the distance from $x$ to $c$ is ...
- 2,987
2
votes
1
answer
710
views
For which clustering algorithms is the Gap statistic useful?
How can i know for which clustering algorithms (with a parameter that represents number of clusters) it makes sense to use the Gap statistic? I've read in the paper by Tibshirani, Walter & Hastie ...
- 451
1
vote
0
answers
19
views
Question about running k means cluster analysis
In a previous analysis I had 3 groups of subjects - group x with 35 subjects, control group y with 25 subjects, and control group z with 25 subjects. For each group I have levels of 6 different ...
1
vote
0
answers
40
views
Question on using the elbow method for calculating ideal number of clusters for k means cluster analysis
Newb to cluster analysis here. I have a group of 35 subjects. For all of the subjects I have data for different measures of IQ (verbal, math, etc) and different biomarkers. There are 6 IQ measures in ...
1
vote
0
answers
22
views
Spatial Temporal Clustering evenly spaced over time
I have a large dataset of spatio-temporal data. It has longitude and latitude coordinates, and a date for each observation. For example:
Long
Lat
Date
50
20.43
9-19-2010
51
19.5
10-4-2010
51
19.3
...
- 121
1
vote
0
answers
125
views
k-means clustering on a probability distribution instead of a dataset
Normally, clustering algorithms such as $k$-means are defined on a dataset in the following sense: if $D$ is a dataset, find a partition of $D$ into sets $\{S_1, \dots, S_n\}$ that minimises the ...
- 111
1
vote
0
answers
68
views
K-means clustering - weird PCA visualization
I performed PCA on 4 variables and are shown in this visualization:
At first look it doesn't look convincing and the some clusters seem weird.
The data was cleaned and standardized beforehand. Only ...
- 11
1
vote
2
answers
388
views
Interpreting results of K-means after PCA
I have this dataset about an airline company customers with 22 explanatory variables. My goal is to perform some sort of customer segmentation with the k-means algorithm. One problem that I've found ...
1
vote
0
answers
48
views
Can K-means put most of the noise in the same cluster?
I am working on clustering text data (very short sentences) vectorized with tf-idf. The data are characterized by high sparseness and the presence of abundant noise (considered here as documents that ...
- 11
1
vote
0
answers
44
views
Differentiate between two set of points
Consider two sets of points (in the pictures below), whose "center of gravity" is same. What measure can differentiate between the two sets?
e.g.
Image 1 ...
- 161
1
vote
0
answers
99
views
Statistical method for finding homogeneous groups of curves
I need to divide a set of 100 or more response curves into groups.
These curves are formed by backscattering intensity along a range of frequencies. Basically, each curve represents the intensity in ...
- 11
1
vote
1
answer
346
views
Clustering Data with Time and ~10 million records
I have a dataset with features like product categories, their dimensions, price, units sold on a given day.
I want to create clusters out of this dataset (~12-15 million records) and I am using data ...
1
vote
0
answers
593
views
What does it mean by "variance of the distribution is spherical"?
I came across the post here which mentioned the following:
k-means assumes the variance of the distribution of each attribute
(variable) is spherical
I wanted to understand what does it mean by &...
- 461
1
vote
0
answers
102
views
Correlated variables and similar loadings in first principal component
I'm doing a K-Means model for first time, thus very low experience. I read that it is not bad to discard variables through some PCA analysis. After standardizing the data, the loadings (weights) for ...
- 113
1
vote
0
answers
38
views
Does it make sense to use variables' product as a new variable in a clustering procedure?
I'm trying to separate different groups based on values from width and length using k-means and hierarchical clustering. My question relates to the possibility of using the area — measured as width * ...
- 111
1
vote
0
answers
58
views
K means clustering analysis to define geological facies using 2 attributes (ERT & seismic)
dear all.
Currently I am doing a project where the goal is to define geological facies of an area by using ML. The method that we are doing is k-means (we have no labels beforehand) and we are using ...
- 11
1
vote
0
answers
82
views
Clustering large yearly, (presence/absence) dataframe
I have a data frame of 500,000x23 dimensions. The data is binary, representing presence or absence. The data follows identified trees through time (23 years) and looks at whether the tree is present ...
- 11
1
vote
0
answers
69
views
Non-negative matrix factorization clusters
NMF can be used for clustering i.e., $V=WH$ where $W$ represents cluster centers and $H$ represents the membership of samples. But can NMF alone cluster the samples? Can we get better clusters in NMF ...
- 329
1
vote
0
answers
579
views
Difference between Hamming Loss, Hamming Score, and Hamming Distance in multiclass multilabel classification
I am trying to understand the mathematical difference between, Hamming distance, Hamming Loss and Hamming score. I am trying to perform two actions
Multiclass multi label classification using SVM
K ...
1
vote
0
answers
134
views
K-Means clustering technique for monthly data
I have an Unsupervised problem where user's Credit Card payment data is given for each month for various users for one year. One of the feature in the data having "User Id". For most of the ...
1
vote
0
answers
160
views
External loss functions for Spectral/Density-based clustering
In this article, Abou-Mustafa and Schuurmans proposed a method that makes it easy to decide what unsupervised learning algorithm generalizes 'better' to the entire dataset. In particular, this needs ...
- 11
1
vote
0
answers
50
views
Do I correctly apply hierarchical clustering and K-means on the resource-selection-function values?
I'm trying to find the best way to classify bivariate point patterns in spatstat according to the relationship between two point species:
Point pattern ...
- 21
1
vote
0
answers
1k
views
k-means inertia
I use Minkoski distance to measure distance, like so:
$$D(\vec{x}, \vec{y})=\left(\sum_{i=1}^n|x_i-y_i|^p\right)^\frac{1}{p}$$
I'm trying to locally optimize centroids by averaging the points that ...
- 111
1
vote
1
answer
160
views
Discussing validity of tests performed after a cluster analysis
I'm new to datascience (from a medical/medical science background).
My supervisor (social sciences background) asked me to assist in rewriting a paper where we do a cluster analysis for a ...
- 11
1
vote
0
answers
172
views
I don't understand why each time Kmeans finds the same centroid given different initialization?
The k-means algorithm does the following:
Given a set of points, we first choose k random points to be the initial centroids.
We then create k clusters. The ith cluster contains the points nearest to ...
- 11
1
vote
0
answers
50
views
Precise definition of K-means
I am reading about K-means algorithm, and trying to explain it to myself in one sentence. However, I am a bit confused. I have came up with following definitions and I am not sure whether which one is ...
- 11
1
vote
0
answers
21
views
Clustering - Distance Metric for Comparing Short Lists of Terms (non-repeating, no frequency)
Clustering involves using some distance or similarity metric.
What is the best way to score the similarity of these small sets of words? Criteria: These are technical terms which are extracted from ...
- 363
1
vote
0
answers
313
views
How to compare consistency between clustering results and list of values with different levels in R?
I found similar subjects on the website but I may have missed the relation with my own question. I'v seen questions about comparison of clustering results, but here it's more about comparing two lists ...
- 11
1
vote
0
answers
44
views
Clustering common words for objects
I am currently running experiments aiming to simulate information transfer between agents. Without going into too much irrelevant detail, following the conclusion of a simulation I am left with a csv ...
- 21
1
vote
0
answers
16
views
Prerequisites/Checks for performing clustering
What are the checks that should be done on our data before performing clustering? Like how to check whether the dataset contains clusters of equal size/density or the clusters present in the dataset ...
- 51
1
vote
0
answers
79
views
What is meant by 'bad clustering' in k-means ++ paper?
Arthur and Vassilvitskii, 2006 say that:
"There are many natural examples for which the algorithm generates arbitrarily bad clusterings (i.e., $\frac{\phi}{\phi_{OPT}}$ is
unbounded even when $n$ ...
- 321
1
vote
0
answers
597
views
Identifying inflexion point in elbow method (cluster analysis)
I am looking for the optimal number of clusters to conduct a cluster analysis and used the following code to determine it:
...
- 377
1
vote
0
answers
40
views
K-means clustering and temporal aspects
I was looking at an interesting case concerning popular locations for
uber, see here:
https://mapr.com/blog/real-time-analysis-popular-uber-locations-spark-structured-streaming-machine-learning-...
- 245
1
vote
0
answers
36
views
Finding centroids without K-means
The Data:
Currently I have a simple distribution that looks like the histogram below. Every point is an integer between 0 and 16 and I have 350MB of these samples.
The Problem:
I want to identify 8 ...
- 131
1
vote
0
answers
318
views
Deterministic Methods to Initialize K-Means and K-Medoids Clustering Methods
I am looking for effective and deterministic methods to initialize K-Means and K-Medoids algorithms.
There is a great answer in Methods of initializing K-Means Clustering yet most of them has some ...
- 1,157
1
vote
0
answers
68
views
Anomaly detection using Clustering techniques on data with skewed features
I am working on finding anomalies in data. My dataset consists of 2 variables - Price of item and Volume of item purchased in a single checkout. I do not have information on the item other than its ...
- 11
1
vote
0
answers
327
views
Strange data point clustering in k-means
I am trying to explore weather impact on bike usage using k-means clustering. However, the plot does not look correct; visually, some data points obviously belong to another cluster that is closer ...
1
vote
0
answers
94
views
How do I evaluate a K-Means unsupervised anomaly detection approach?
how do I evaluate K-means clustering anomaly detection method as there is no labelled data of anomaly class. To find the cluster (K), I have used the silhouette score from Scikit learn library. Scikit ...
- 11
1
vote
0
answers
162
views
python find the optimal # of cluster for K-Means algorithm
I have a data that contains 24 features and all features have some missing values. I want to use the impute-KNN algorithm from sklearn to fill the missing values. However, before I do that, I think I ...
- 21
1
vote
0
answers
19
views
K means clusters, would PCA be a better option?
I have the data below. I need to use a clustering method to classify them and into categories of "Heterozygotote, Allele 1, Allele 2 and No Call. The values in RFU1 and RFU2 are used to determine the ...
1
vote
1
answer
360
views
Should I scale when clustering text data using K-means?
I want to cluster a folder of texts. I created a data file where for each text, I write whether a certain word appears in it or not. I want to cluster according to this.
So my matrix is globally only ...
- 363
1
vote
0
answers
177
views
K-Means Unable to Detect Small Clusters
I am just wondering about this issue brought up by our teacher about a drawback of K-means being unable to detect small clusters. It's homework that we should come up with ideas about why this is so ...
- 43
1
vote
1
answer
151
views
Modifying k-means for points on torus
My data coordinates are degrees so each axis has values [-180, 180]. Therefore it's easy to spot that in fact the scatter plot on the right end continues on the left side and the same thing for up and ...
- 61
1
vote
0
answers
221
views
How to search for irregular signals: Fourier, DWT or k-means?
See my notebook here
I want to search for irregular time signals in a data set of ~3 500 000 time signals. I can't give a clear definition of irregular signal, but it must fulfil the criteria of: not ...
- 121
1
vote
0
answers
532
views
Why does kmeans after SVD result in ideal clusters
I am clustering tweets which are related to eye fashion and they are extracted using keywords like mascara, eyeliner, eyeshadow, etc from twitter.
I
constructed a Tf-idf matrix (tweets x words) ...
- 163