All Questions
Tagged with cluster or clustering
4,041 questions
0
votes
0
answers
38
views
Adjusted TF-IDF where many terms appear in every document
Struggling with something so hoped the brilliant minds of the internet could help me out.
I have a large dataset of job postings from which I have extracted the skill demand (no. of times a skill is ...
- 3
2
votes
1
answer
367
views
When two distributions overlap, how to separate one distribution from the mixture distribution if I the other distribution is known?
I have a data which can be classified into two groups. As you see, Figure(a) shows that they are easily classified into group A and B. However, sometimes they are overlapped and it is impossible set a ...
1
vote
0
answers
28
views
Analyzing binary survey data - after clustering, how move forward in multivariate/ordination analysis?
I am new to multivariate analysis.
I have been given a set of survey answers with around 120 binary questions. My goal is to first perform cluster analysis to find the optimal number of respondent &...
- 11
1
vote
0
answers
209
views
Elbow plot and plot of average silhouette width disagree with each other
I am fairly new to using clustering. On the data science course I am on, we recently covered agglomerative clustering and k means clustering. I have created a toy example to see if I can use R to ...
- 193
0
votes
0
answers
51
views
Clustering algorithm on undefined number of classes
Imagine we have a large system of images. We are trying to search for an image and images that are closer to it. One strategy I could think of is to embed the image say into a 100-dimensional vector. ...
- 249
0
votes
0
answers
27
views
Conceptual understanding of effect of standardizing after normalization for clustering
My general understanding is that, before running a clustering algorithm, one typically wants to consider trying to normalize or standardize the data depending on its content and use. In my case, I ...
- 308
1
vote
0
answers
73
views
Methods describe the temporal consistency of kernel density data
I am working on a spatial time series analysis project. The task is to study the spatial distribution of point features (e.g., crime events, traffic accidents) over time. I aim to find the places with ...
- 107
1
vote
0
answers
39
views
How useful is PCA on its own? [closed]
In my machine learning course we have covered the key ideas behind principal component analysis. To round this part of the course off, we have learned to interpret the results of PCA, specifically ...
- 11
1
vote
0
answers
87
views
Clustering mixed data - SPSS [closed]
On my current project, I have to form four or five clusters describing different types of banking customers. The data is based on a survey of around 3500 participants and contains more than 250 ...
- 11
1
vote
1
answer
39
views
Statistical argument for this cluster measure
This quick clustering score discussion presents the following single cluster scoring functions:
$$
c_i = 1-\sqrt{\frac{\sum_{j}^{N}\left(1-\phi(i,j) \right)^2}{N-1}}
$$
and
$$
C = 1-\sqrt{\frac{\sum_{...
- 123
2
votes
2
answers
54
views
Clustering a single binary variable?
I am trying to design a study which will involve asking people to watch videos and then stop them at points that they think relevant (looking for specific behaviours). The idea is to see if people ...
- 21
2
votes
1
answer
445
views
Latent Class Analysis - Interpretation and integration with survival analysis?
I am approaching to Latent Class Analysis to identify "classes" of patients based on some variables.
Question 1: diagnostics of the results. I already gone through the discussion on whether ...
- 706
0
votes
1
answer
59
views
Analysis to identify groups of occupations with similar skill demand from job postings?
Was told to post this here so hope it's the right place!
I am using a very large dataset of job postings, which for each posting has a unique posting ID, the job posting occupation, and a row for each ...
- 3
0
votes
0
answers
58
views
How to identify the clusters in SSE plot?
How to determine the number of clusters from the following plot?
- 1
2
votes
0
answers
13
views
How to tell if data can be used as a predictor in a classification or clustering model?
I'm currently studying a data science handbook in preparation for job interviews, and I came across a question that I think should be simple, but honestly has me stumped. Here's a screenshot of the ...
- 167
4
votes
1
answer
2k
views
How to compare labels from clustering analysis and original ones?
I was asked to run a clustering analysis to assess the validity of labels for a manually labelled dataset.
I can simply save the actual labels (4 classes: 0, 1, 2, 3) and run clustering analysis (let'...
- 127
0
votes
1
answer
49
views
Clustering and coordinate rotation
Does the coordinate system rotation affect the clustering result?
Which approach could be used to eliminate the influence of coordinate system rotation in clustering?
Any help would be appreciated！
- 61
1
vote
0
answers
17
views
truncated Functional data analysis + identify subgroups?
I have intensive longitudinal data about heartbeat rates, movement activity measured by actiwatch, respiration rates etc... collected from about 100 patients at a terminal care.
For each patient, data ...
- 21
1
vote
2
answers
120
views
How to cluster and visualise vectors of which the components are class indices?
Let's say I have a dataset $\boldsymbol{\mathcal{X}}$ of $N$ samples wherein each sample $\boldsymbol{x}^{(i)}\in \mathcal{X}$, $i \in {1 \ldots N}$, is described by a set of $D$ features, such that $\...
0
votes
0
answers
57
views
Spectral clustering implementation in python yields nonsensical results
I am trying to implement spectral clustering in python. I'm using these excellent notes as a guide. As the notes suggest, I'm using a toy model where I generate random numbers that come from ...
- 141
1
vote
0
answers
200
views
Linear Distance in Latent Feature Space of an AutoEncoder
I would like to perform a cluster analysis on a mixed data set containing continuous, categorical and binary data.
As I have 93 features in total, I thought it might help to use an AutoEncoder to
...
- 11
1
vote
0
answers
35
views
Any other machine learning algorithms besides clustering for Customer Segmentation
I've searched online materials for Customer Segmentation for a while. Most materials are related with K-means or some other clustering algorithms. Are there any other machine learning or statistical ...
- 519
0
votes
0
answers
31
views
Clustering zip data at state level
Problem : Recomment 2 markets (States) for A/B test
Solution tried :
Cluster rows based simply by ZIP codes but then each state could appear across multiple clusters. How do I then dedup the results?...
2
votes
0
answers
37
views
Does it make sense to transform a feature containing hours (24h) into two features with xy-coordinates of each hour in the space? [duplicate]
I have a clustering problem that I might solve with an algorithm based on Euclidean distance (e.g. K-Means).
One potential feature is the "hour" at which each user began an interaction.
As ...
- 566
1
vote
0
answers
162
views
Why cannot I use silhouette score with ground truth labels?
I was looking into checking cluster positioning from a non-liner transformation. I do have the ground truth labels. Now, I want to use the transformed data points and see how good this transformation ...
- 111
2
votes
0
answers
82
views
Clustering algorithm that penalizes the difference in cluster size
Does anyone know a clustering algorithm that penalizes the difference in cluster size (= number of elements)?
If so, is it available in any package (python, R...) ?
For example, the following ...
- 121
2
votes
0
answers
263
views
Clustering on MDS Data
I have computed a matrix of MDS distances using R's dist() function and then reduced to two-dimensional coordinates using cmdscale() function. If I apply PAM or k-means clustering (the choice of ...
- 31
2
votes
1
answer
102
views
In DBSCAN, what happens if points have distance exactly equal to the Epsilon radius of a core point?
In DBSCAN the border points are points in the eps-neighborhood of a core point. But what if a point has distance exactly equal to Epsilon from a core point? Is it considered inside the eps radius, or ...
- 43
1
vote
1
answer
342
views
What is a good way to cluster a variable length sequence data
I have a dataset consisting of variable length sequences: There are about 5000 instances. Each instance is a variable length sequence. The shortest sequence has a length of 2 and the longest sequence ...
- 111
2
votes
2
answers
90
views
Curve quantification
I have some longitudinal measurement data of 15,000. I smoothed that data with B-spline smoothing and got the following curve.
I then want to quantify this curve and extract features for clustering ...
- 35
0
votes
0
answers
31
views
How to interpret the Scatter Plot result from PCA? [duplicate]
I have a project in school about clustering analysis. I have applied standardization and principal component analysis (PCA) to my dataset (I used K-means), which is about heart disease patients. I ...
- 1
1
vote
1
answer
140
views
In $k$-means, how is it NP-hard if the dimensionality of the data is at least $2$ ($d\geq 2$)?
In $k$-means, how is it NP-hard if the dimensionality of the data is at least $2$ ($d\geq 2$)? Can someone justify or give reasons to this statement?
Any guidance would be appreciated.
10
votes
1
answer
425
views
Are there algorithms to cluster Graphs, not just cluster nodes in a graph?
I am wondering if there are algorithms to cluster graphs; what I meant is to cluster many graphs, not cluster the nodes in a graph.
For example, we have N graphs, G1, G2, G3, .....GN. Then we can ...
- 387
7
votes
5
answers
4k
views
Would a machine learning classifier algorithm be able to determine whether a number is odd or even?
I was testing out some classifier algorithms in scikit but wasn't able to find a classifier (linear or non-linear) that managed to provide good prediction on whether an input number is odd or even. ...
- 189
0
votes
0
answers
205
views
Negative gap statistic interpretation for cluster analysis
I am trying to perform a cluster analysis on a dataset. The plot of clusters vs. gap statistic is below. I do not know how to interpret the decrease of the gap statistic and its values which are ...
- 23
1
vote
0
answers
19
views
References about clustered linear regression
I have to write a project about clustered linear regression.
The only resource I have been able to find is https://doi.org/10.1016/S0950-7051(01)00154-X
Any other articles/books that you can point me ...
- 11
0
votes
0
answers
28
views
No gaussian shape after log transformation and scaling
I have customer purchase history data which has variables like recency, frequency and ...
- 3,342
1
vote
0
answers
27
views
Hypothesis testing for a relationship, on a subset of my data
Suppose I have a collection of paired samples $(X_i,Y_i)$, where $X_i\in\mathbb{R}^n$ and $Y_i\in\mathbb{R}$. We find that $X$ and $Y$ are not particularly correlated with one another, but when we ...
- 139
0
votes
0
answers
22
views
K-means on linearly projected features
I am looking for references on K-Means applied to linearly projected features instead of to the original features, in the sense that both K-Means and the projection matrix are learned at the same time....
- 213
1
vote
0
answers
48
views
Can K-means put most of the noise in the same cluster?
I am working on clustering text data (very short sentences) vectorized with tf-idf. The data are characterized by high sparseness and the presence of abundant noise (considered here as documents that ...
- 11
2
votes
0
answers
51
views
How can I ensure both levels of a binary variable are represented in every cluster?
Let's say I have some continuous variables and a binary treatment indicator. I want to cluster my observations based on the variables while ensuring that each cluster contains at least one member of ...
- 36.8k
2
votes
0
answers
370
views
How does this prove that the objective function in K-means clustering never increases?
I am reading the ISLR textbook (pg. 518-519, 12.4) and having trouble understanding why K-means clustering never increases. I can understand it conceptually but I don't understand the mathematical ...
- 21
0
votes
0
answers
30
views
Cluster a set of files by the the number of points
I have a large set of aerial images with herds of elephants in it. The number of elephants in a single image can range from ~ 20 elephants to 1.
I have created a dataset of ~ 2,000 png image files ...
- 31
0
votes
0
answers
2k
views
Silhouette Score with Noise (from DBSCAN)
I stumbled across this example on scikit-learn (1.2.0), where the silhouette score alongside some other metrics is computed for DBSCAN cluster assignments. These assignments include some Noise ...
- 419
0
votes
0
answers
24
views
What statistical methods can I use to determine the popularity of item combinations taken from two groups?
Let's say I have two tables, listing projects with the programming languages and frameworks they use:
...
- 1
1
vote
1
answer
123
views
Pair confusion matrix for more than two cluster algorithms
How would one code the following issue:
The columns represent the k groupings from the k cluster algorithms. The rows are the N(N-1)/2 pairs of data points. If an algorithm groups a pair together (i,j)...
2
votes
2
answers
1k
views
Time series clustering on large data
I am trying to do K-means clustering on my data which has time series length of 3700 and for (latitude,longitude) points of around 6000 in length. However, timeseries clustering using tslearn package ...
0
votes
0
answers
115
views
What are consequences of forming clusters on variables that are not normally distributed?
I am trying to use DBSCAN to obtain clusters of chess player rating changes (Elo rank) over one year of games.
I have a bunch of input variables, some of which are not normally distributed even if I ...
- 23
3
votes
2
answers
764
views
Is 'High School', 'Graduate', 'Unknown' ordinal or nominal data?
My goal is to Feature Engineering the column Education_Level. This is an obvious ordinal data. However, I am having difficulty to put Education_Level to choose <...
0
votes
0
answers
81
views
overlapping clusters in R
I have a bunch of observations that are correlated with each other in two different ways, so I'd like to perform logistic regressions but try to account for the correlations among observations to get ...
- 3