Questions tagged [hierarchical-clustering]
Hierarchical cluster analysis is a method of cluster analysis which builds, by steps, a hierarchy of clusters, a dendrogram. Most popular is agglomerative hierarchical clustering (HAC) which starts from individual objects and collects them into bigger and bigger clusters.
481
questions
1
vote
0
answers
20
views
What statistical tests can I use in R to ascertain the statistical significance of clusters obtained using DIANA (DIvisive ANAlysis Clustering)?
I have a data set of cuticular hydrocarbons (CHCs) from 60 samples of aphids collected across mid, early and late activity season on three different plants species that they fed on. The CHCs can ...
- 11
0
votes
0
answers
10
views
How to offset, align and increase padding of dendogram labels [migrated]
I am trying to create a dendogram in R. As of now, I have used the factoextra package, and specifically the fviz_dend function. The code is as follows:
...
- 11
0
votes
0
answers
12
views
Statistical techniques to compare leaves position across ordered trees
I want to tackle this problem.
Suppose I have a set of entries that can produce multiple ordered trees based on some statistical methodology. The entries are always the same, but the tree structure ...
- 445
1
vote
2
answers
32
views
About adding random effects in Multilevel (HLM) analysis
I am doing regression analysis in HLM. I am wondering whether random effects should be added in this process.
Let me ask a question using a famous example. LV1 is a student and LV2 is a school. LV1 ...
0
votes
0
answers
12
views
calculation of the C-index clustering for manual [duplicate]
Can anyone give me an example of working on the C-index clustering validity test, but calculating manually??
- 1
0
votes
0
answers
25
views
calculation of the C-index clustering [duplicate]
Can anyone give me an example of working on the C-index clustering validity test, but calculating manually??
- 1
0
votes
0
answers
19
views
Hierarchical Clustering Using Mutual Information
I am interested in Hierarchical Clustering Using Mutual Information. Asking the ChatGpt, I got this:
...
- 292
0
votes
0
answers
20
views
Best clustering algorithm when needing to use a distance matrix
I have about 1300 arrays that are 28x1 in size which contain numerical data.
They tend to have some parabolic shape (when plotted), but certainly not all of them.
I want to see how many 'different' ...
- 1
0
votes
0
answers
9
views
Combining classification models and regression (Hierarchical? or MoE modeling?)
Assume, there is a dataset of human poses captured from different angles. For each sample there are 2 level of labels - pose type (standing, sitting, etc) and pose magnitute(continuous).
Defined tasks,...
1
vote
0
answers
28
views
Evaluate hierarchical clustering with partial ground truth
I am performing hierarchical clustering, and I need to decide which agglomeration method to use.
While I don't have a ground truth, I know that some datapoints should be closer together: for example, ...
- 167
1
vote
0
answers
21
views
Hierarchical Clustering with Large Datasets
I am currently aiming to perform hierarchical clustering for the purpose of customer segmentation. My dataset consists of 217,000 instances with 12-15 features. However, due to memory issues when ...
- 21
1
vote
1
answer
31
views
Name for a dendrogram showing how variable clustering affects variance explained
This is from Harrell's RMS:
I'm sure I have seen a similar dendrogram used to illustrate the effect of variable clustering on $R$ / $R^2$. It was plotted with the x, y axes the other way round to the ...
- 929
0
votes
0
answers
15
views
Is there some sort of multilevel KNN/ML model I can use to figure out which users will buy specific products?
I am wondering if there is some sort of multilevel model that I can use to identify likely buyers of specific products or create a lookalike audience.
The issue is that I have 1000s of products and ...
- 65
1
vote
1
answer
51
views
Hierarchical clustering of a distance matrix with element weights
I am computing a hierarchical clustering of some geospatial data. I need to add in an element weighting to the approach.
My current approach is:
I compute temporal cross-correlations between my N ...
- 51
2
votes
1
answer
51
views
How to interpret hierarchical clustering results
I have a data frame "Customer_original" with customer data, it has around 50 mixed type variables.
I standardize "Customer_original" and put the standardized values into "...
1
vote
0
answers
59
views
How to incorporate p-value information for pairwise distances into clustering?
Consider I have a $n \times n$ distance matrix I want to use for hierarchical clustering, where the distance metric I use ranges from 0-1.
I also have a second $n \times n$ matrix that gives me the p-...
- 89
0
votes
0
answers
27
views
Hierarchical cluster analysis with mixed data
I have a dataset consisting of 134 observations and four variables. The dataset consists of answers to a questionnaire. I want to perform a hierarchical cluster analysis on the variables in the ...
- 1
0
votes
2
answers
164
views
Best cluster approach for dichotomous variables in SPSS
I'm reading contradictory information on the internet and wanted to get some thoughts on a statistical approach to use. I'll use a fictional example for the sake of easy interpretation.
I have a set ...
0
votes
0
answers
47
views
Drawing a dendrogram based on a confusion matrix
I have a simple 2D confusion matrix that I normalised for every true class. Now, what would be the best method to visualise this as a dendrogram?
I have seen hierarchical agglomerative clustering on ...
2
votes
1
answer
51
views
Understanding complete linkage
I was trying to understand the linkage function from scipy and I was confused with the output for this sample code
...
- 123
0
votes
0
answers
64
views
How to determine which variables are the most important to define each cluster from a Hierarchical clustering analysis, using R
After searching, reading posts, and trying to solve this issue myself for several days, I think it's time to ask for help.
I'm performing a Hierarchical Clustering analysis on a dataset, that includes ...
- 1
0
votes
0
answers
31
views
How to perform Hierarchical Clustering using centroid method and custom distance metric?
I would like to perform Agglomerative Hierarchical Clustering using the centroid method (defined on this page) and a custom distance metric, probably cosine similarity. In the Scipy docs it says you ...
- 61
0
votes
0
answers
28
views
Bayesian Hierarchical Clustering prior update
I am working through Heller and Ghahramani's "Bayesian Hierarchical Clustering" paper (https://www2.stat.duke.edu/~kheller/bhc.pdf) and things aren't quite working out the way I expect with ...
1
vote
0
answers
40
views
Why is HDBSCAN considered a transductive clustering algorithm despite having approximate_predict?
In the documents for HDBSCAN it says the following:
Often it is useful to train a model once on a large amount of data,
and then query the model repeatedly with small amounts of new data.
This is ...
- 23
0
votes
0
answers
22
views
How to model clustered data structure association when all clusters are observed at the same time?
I have data from many recipes I made that describes how different ingredients subgroups grouped in different parent categories (sweet, sour, bitter, umami ) work together to ultimately lead to scores ...
0
votes
0
answers
28
views
Peculiar silhouette score is reduced with more clusters
I am working with Python and a set of sensors. I want to use clustering on them.
To do that I have used hierarchical cluster with average linkage and euclidean distance as:
...
- 205
1
vote
0
answers
47
views
Clustering for a meaningful fixed effects model
I have panel data for 5 time periods and about 10,000 geographic units.
My dependent variable Share is the share of workers in a specific category, and my descriptors represent various factors that ...
- 97
1
vote
1
answer
41
views
Why is the dendrogram structure different from what I see of my data in a 3D PCoA?
I conducted hierarchical clustering on a dissimilarity matrix computed from a series of species distribution models text and I seem to get different visible structure between the dendrogram and the 3D ...
- 31
4
votes
2
answers
376
views
Question about Silhouette index calculation using scikit
I am currently working with continuous data measured from different sensors (thermometers and voltmeters). I have a matrix whose columns represent the sensors and the rows are normalized measurements (...
- 205
0
votes
0
answers
25
views
Discrepancies between AU and BP values in HCA
I conducted a hierarchical cluster analysis in R on a pool of 20 words that have certain common traits. I used hclust(). After, I did multiscale bootstrap resampling using pvclust(). While smaller-...
- 21
0
votes
0
answers
15
views
Alternatives to spatial and temporal aggregation of time series to discover more learnable patterns
Given taxi demand time series of towns in a country. I would like to do demand forecasting.
I noticed that when the town's time series is zero inflated the prediction is poor. However, when these ...
- 294
0
votes
0
answers
16
views
Clustering of Variables around Latent Variables (CLV) over only qualitative data
I'm reaching out today because I have a concern regarding the clustering approach employed with the CLV method introduced by Vigneau and Qannari in 2003. I've noticed that this method is predominantly ...
- 21
0
votes
0
answers
20
views
Design Effect in Cluster Randomised Trials
I want to understand why and how variance inflation factor is taken in account with the formula [1+(m-1)*ICC], where M= Size of clusters and ICC is the intracluster correlation.
As I have deduced till ...
0
votes
1
answer
117
views
Multivariate Longitudinal Multi-level Model: What is the hierarchy structure?
I am interested in running a multivariate multi-level model with longitudinal data, and I'm having a hard time conceptualising the hierarchy levels.
My variables are:
Multivariate outcome: Quality of ...
- 1
4
votes
1
answer
153
views
Combine two estimates of same variable
Suppose I know that $ A = 0.3 \cdot B + 0.7 \cdot C $
And I have these estimates:
$$ \hat A = A + \varepsilon $$
$$ \hat B = B + \zeta $$
$$ \hat C = C + \eta $$
For the sake of the argument, $ \...
- 63
0
votes
1
answer
34
views
In unsupervised learning, is a result of 2 clusters meaningful?
I used both agglomerative clustering and k-means on a dataset and see the results below. Result from agglomerative clustering was demonstrated with silhouette score while kmeans with inertia score. ...
- 189
2
votes
1
answer
77
views
Inconsistent cluster indices using hierarchical clustering for time series data
I am currently trying to spatially cluster data that is ordered on a grid. Each point has x and y coordinates as well as a measurement value. These features come from a time series where I analyze ...
- 167
0
votes
0
answers
382
views
How to automatically choose the optimal number of clusters for a hierarchical clustering dendrogram? (R or Python)
I have a product substitutability matrix, where each cell has a number between 0 and 1 to represent how substitutable (i.e. how similar) the two products are.
I then apply hierarchical clustering to ...
- 1
1
vote
0
answers
53
views
Finding the nearest point to a given set of points [closed]
I am trying to build an auto-suggestion logic which looks at currently selected items and recommends a list of items to select the next item from.
I can formulate the problem thus:
Given a set of ...
1
vote
0
answers
90
views
Cluster analysis after factor analysis: What distance measure to use?
I use factor analysis on a set of 15 survey questions (likert scales).
Using the predict command (in stata) I make 5 factors.
Subsequently, I want to use cluster analysis to see if there are "...
- 11
1
vote
0
answers
312
views
What are the downsides of using euclidean distance for hierarchical clustering of a correlation matrix?
Apologies if this has been answered elsewhere, but I couldn't find any answers discussing this specific question.
I am lacking some notion on clustering using euclidean vs correlation distance, when ...
- 11
1
vote
1
answer
261
views
Unsupervised learning: How to identify differences between clusters?
I'm learning about unsupervised learning and I tried to use KMeans, AgglomerativeClustering and DBSCAN on the same datase. The result was ok, they seems to work fine according silhouette_score() ...
2
votes
1
answer
337
views
Choosing the best clustering algorithm and evaluating the results
I'm trying to separate my data into clusters using the k-means algorithm and the hierarchical algorithm, choose which algorithm fits my data the best, and evaluate the results. However, all of my ...
- 61
10
votes
1
answer
414
views
Are there algorithms to cluster Graphs, not just cluster nodes in a graph?
I am wondering if there are algorithms to cluster graphs; what I meant is to cluster many graphs, not cluster the nodes in a graph.
For example, we have N graphs, G1, G2, G3, .....GN. Then we can ...
- 387
0
votes
0
answers
176
views
Negative gap statistic interpretation for cluster analysis
I am trying to perform a cluster analysis on a dataset. The plot of clusters vs. gap statistic is below. I do not know how to interpret the decrease of the gap statistic and its values which are ...
- 23
0
votes
1
answer
40
views
Feature Scaling in Hierarchal Clustering
I know that feature scaling is always a requirement for clustering algorithms. Currently I am implementing hierarchal clustering on this dataset, I will use only the annual income and the spending ...
- 145
3
votes
1
answer
897
views
Using Ward's method on a dissimilarity matrix of Gower distances
I have a question regarding Wards method of hierarchical clustering.
I used Gower Distance to create a dissimilarity matrix from an event log. I want to agglomerate it with Ward's method.
Lets suppose ...
- 133
0
votes
1
answer
59
views
What graph is best suited for this data? (and how can I produce it in R)
I have this df object (9x19),
...
0
votes
0
answers
124
views
Clustering Algorithm with Continuous and Binary Variables
I am creating a clustering analysis with both Continuous and Binary variables and am wondering
What type of model is best for these cases
How should I scale my data. Should I only scale Continuous ...
- 11
1
vote
1
answer
76
views
Distance function that captures both circular and "appear as line" clusters [closed]
based on what I know in k-mean clustering, if i use single linkage distance it can capture clusters of thread shapes but it is not suitable for capturing circular clusters.
Also If we use complete ...
- 11