Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
704
questions
1
vote
0
answers
23
views
Derivation in Gap Statistic
I am reading the paper "Estimating the number of clusters in data set via the gap statistic" by Tibshirani (2001), and there is a formula that I do not see how it is derived. It says
$$
E[\...
- 11
1
vote
0
answers
36
views
What if PCA is unable to group my samples, but K-means perfectly clusters them? Is there any problem with my data analysis? Is it possible? [closed]
I am not an expert, but I am currently using unsupervised methods to better explain my mass spectrometry data obtained via DART-MS analyses. I am still learning.
It turned out that when analyzing my ...
- 11
0
votes
0
answers
7
views
Comparing unsupervised approaches to topic prevalence detection in semi-structured nested textual data
I am trying to understand approaches to (ideally, probabilistically) detecting the prevalence of latent topics in semi-structured, nested textual data. Specifically, there appear to be at least two ...
- 859
1
vote
1
answer
22
views
Loss functions for unsupervised clustering of gaussian data
I got data that was generated from $n$ multivariate gaussians with different means and covariances, which means that they are separateble, and I'm tasked with classifying them using neural network in ...
- 157
2
votes
0
answers
14
views
Unsupervised clustering of short texts with covariates
I posted this on the Data Science Stack Exchange and didn’t get any responses (that sight seems pretty dead). So I’m trying here!
I'm working on a project where I have to categorise short texts. I don'...
- 45
0
votes
0
answers
29
views
How to find multi-variable correlation?
I have a degree in engineering but no deeper knowledge in statistics so I'm wondering if anyone can point me the direction to find a solution to the following problem:
I have a set of input variables ...
- 101
3
votes
0
answers
65
views
How to Automatically Identify Column Headers in New Datasets Using Machine Learning?
I have a dataset with vehicle data that includes headers (example):
I receive new datasets with similar vehicle data but no column headers:
I need to create an algorithm to recognise and assign ...
- 31
3
votes
2
answers
78
views
Can an Anomaly Detector be Tested with Data that it Labeled?
Is it wrong to leverage a model to label data, then perform a train/test split to evaluate the performance of said model?
Assume I have an unlabeled data set where the missing labels are a binary ...
- 145
2
votes
1
answer
50
views
Interpreting Mass-Volume as an evaluation criterion for unsupervised anomaly detection
I have found this paper How to Evaluate the Quality of Unsupervised Anomaly Detection Algorithms? by
Nicolas Goix that talks about evaluation of unsupervised anomaly scoring functions by the use of ...
- 243
0
votes
0
answers
28
views
How can I integrate time in my Implicit Feedback dataset?
I'm working on a recommendation system based on Collaborative Filtering. Specifically, I've been looking at models such as NCF (Neural Collaborative Filtering) and SAR (Simple Algorithm for ...
- 1
2
votes
1
answer
55
views
Should we routinely conduct unsupervised learning when reporting descriptive statistics on data?
A standard approach prior to conducting a predictive or inferential analysis is to report some basic univariate descriptive statistics on the study variables: mean, median, minimum, maximum, variance, ...
- 6,174
0
votes
1
answer
9
views
How to Define Equipment Churn in Laboratory Service Data Without Explicit Churn Labels?
I'm working with a comprehensive dataset spanning 20 years of service records for laboratory equipment owned by various customers. This dataset captures intricate details, such as the equipment ID, ...
- 1
0
votes
0
answers
9
views
What is meant by the assumption of statistical independence among sources in Independent Component Analysis?
One of the underlying assumptions of independent component analysis (ICA) that I consistently see written is "statistical independence across the source signals". In the context of the ...
- 71
0
votes
0
answers
12
views
ML Clustering with an added condition
Problem: I want to create distance-based clusters of customers where each cluster, in sum, yields the same revenue potential.
Explanation:
I'm looking at thousands of customers spread throughout a ...
1
vote
1
answer
72
views
Using unsupervised methods prior to cross-validation when all unlabelled data is available
There is lots of discussion about pre-processing methods and if they need to be included within a cross-validation procedure or if they can happen prior to splitting the data -- questions on ...
- 228
13
votes
4
answers
1k
views
Why does Harrell Argue for "Ignoring Y during data reduction"?
In Regression Modeling Strategies page 79 (4.7 Data Reduction) reads:
Data reduction is aimed at reducing the number of parameters to estimate in the model, without distorting statistical inference ...
- 143
1
vote
0
answers
14
views
Adjusted Rand Index Per Single Cluster
i have a nuanced ARI question for which i can't find any reference.
say i have two clusterings, say ClusteringA and ClusteringB, and i am comparing them using ARI. now, i'm not interested in the ARI ...
- 11
0
votes
0
answers
81
views
How to validate unsupervised anomaly detection in absence of ground truth?
I am currently working on an unsupervised anomaly detection project and facing a challenge regarding the validation of the model's performance due to the absence of ground truth labels.
I am using ...
- 21
0
votes
0
answers
39
views
Difference between Principal Component as closest linear surface and regression line as best linear approximation in two-dimensions
I am currently working on a presentation on PCA as part of a seminar programme and one slide is about giving an interpretation on the first constructed Principal Component on an arbitrary two-...
4
votes
3
answers
308
views
Outlier detection methods aware of target variable
I am trying to predict ambulance demand for the next hour, for a city area in the USA, based on previous demand, weather, large people gatherings, and similar spatio-temporal factors.
I have noticed ...
- 61
0
votes
0
answers
31
views
How to perform Hierarchical Clustering using centroid method and custom distance metric?
I would like to perform Agglomerative Hierarchical Clustering using the centroid method (defined on this page) and a custom distance metric, probably cosine similarity. In the Scipy docs it says you ...
- 61
0
votes
0
answers
40
views
What method(s) answer question 'should X be a category'?
To some degree my problem may be lacking terminology to ask the question well.
Form of a problem I'm looking at is as follows: some number of subjects answered (luckily in a binary way, that is: 'yes',...
1
vote
0
answers
40
views
Why is HDBSCAN considered a transductive clustering algorithm despite having approximate_predict?
In the documents for HDBSCAN it says the following:
Often it is useful to train a model once on a large amount of data,
and then query the model repeatedly with small amounts of new data.
This is ...
- 23
2
votes
0
answers
18
views
Looking for a way to train a model to learn optimal parameters/hyperparameters of clustering
I have 5000 docs, each is a review. For each review, I'm plotting the sentences in a semantic dimension. Now, I'm applying clustering to these points for each review. The success of my model depends ...
- 21
2
votes
1
answer
481
views
When to use Mixture Models (e.g., Latent Class Analysis) vs. Cluster Analysis (e.g., K-Means) for segmenting subpopulations?
I have watched a video describing the differences between Cluster Analysis and Mixture Models.
Clustering determines compact clusters and assigns people to those clusters. Clusters cannot be ...
- 1,123
1
vote
1
answer
94
views
What's the best clustering algorithm for Fraud Data?
Background
I'm working on a Fraud dataset with 500,000 samples, and 130 features.
There are:
98 numerical features,
32 categorical features,
There are missing values in:
7 numerical features,
12 ...
- 645
2
votes
1
answer
61
views
What is the point of finding the best matching unit in a self organizing map?
During the update step while training a self organizing map, the best matching unit for a particular input is found, then that BMU (and to a lesser degree it’s neighborhood) is perturbed to better ...
- 21
4
votes
1
answer
159
views
Examples of unsupervised neural networks that are not self-supervised
I'm trying to understand if such things exist. For the NN to learn it seems that we need a measure of error and thus we need labels. In the case of auto-encodoers, which are considered unsupervised, ...
0
votes
1
answer
87
views
'City climate twins': similarity measure for monthly temperature data
Cities are getting hotter due to climate change. I want to communicate this intuitively through a concept of 'city climate twins': ie. which other city will my city's climate resemble several decades ...
2
votes
0
answers
10
views
How to prove that 2d support vectors are enough for Hard Margin Linear SVM?
As the question states, how can I prove mathematically that 2d support vectors are enough to always be able to formulate the Maximum Margin Hyperplane in d dimensions?
- 21
0
votes
0
answers
12
views
Toy dataset: Radial VAE
I'm evaluating disentanglement in toy datasets seeing as we have such little understanding of the phenomena. I'm using various tools from differential geometry. Now I want to train a VAE on the ...
- 131
0
votes
1
answer
75
views
Deriving a boundary for a single cluster data
Summary
I have a dataset of 2D points that exhibit a distinct pattern. My goal is to create a boundary that encompasses the main cluster of points while excluding outliers. In other words, if a point ...
0
votes
1
answer
34
views
In unsupervised learning, is a result of 2 clusters meaningful?
I used both agglomerative clustering and k-means on a dataset and see the results below. Result from agglomerative clustering was demonstrated with silhouette score while kmeans with inertia score. ...
- 189
0
votes
0
answers
83
views
LMest for cluster membership over time
I have a dataset of 18 continuous variables measured over 3 time points for 90 patients. I hypothesise that there are clusters of patients with similar characteristics and that cluster membership may ...
- 227
4
votes
1
answer
68
views
How is the "training error" of KNN plotted?
On https://www.cs.ubc.ca/~murphyk/Teaching/CS340-Fall07/L4_knn.pdf
Page 6
The author (Kevin Murphy) plots the training and test errors associated with kNN classifer.
I am not sure how training error ...
- 1,590
1
vote
1
answer
255
views
Approximate Gower's dissimilarity measure
I have a very large dataset with mixed-type variables. When I apply the Gower's dissimilarity measure to obtain the distance matrix, it is running out of memory. Due to the large size of the data, it'...
- 41
0
votes
0
answers
46
views
Measuring similarity between two datasets
This is quite straightforward:
I have a dataset with 8 features containing cell measurements from a patient. I want to compare two patients to a reference one. To do this I perform UMAP to reduce the ...
- 13
0
votes
0
answers
89
views
Metrics for multiclass classification
Background
Consider a clustering problem where a dataset of measurements $Z\triangleq\{z_i\}_{i=1}^m$ must be partioned in $n$ clusters, where $n$ is unknown and must be estimated.
Here the term "...
- 383
0
votes
1
answer
10
views
Hierarchical labelling for independent variable in training data
I have data where the objective is to use supervised learning to predict 4 different outcomes.
Say the classes are 1, 2, 3, 4. Though discrete, they are also hierarchical, where class 2 is of an ...
- 101
1
vote
0
answers
34
views
Regression with unlabeled data from several clusters
I want to characterize the relation of a few input parameters to a single output parameter. The problem I have is that my data is collected from several groups. The groups are defined both by the ...
- 205
4
votes
0
answers
110
views
Is there any reason why there appears to be not much modern research into self organizing maps (SOMs)?
Usually there are clear advancements in ML methodologies in research especially, where I can say X method is essentially better than Y method for most datasets.
However, I recently accidentally ...
- 545
1
vote
0
answers
258
views
Unstable HDBSCAN & UMAP clustering results
I've segmented my data using UMAP (dimensionality reduction, with a fixed random seed) and subsequently HDBSCAN to generate a number of clusters.
I've also looped through different values to fine-tune ...
- 11
1
vote
0
answers
19
views
unable to figure out why overlapped clustering is occuring
i am appling k means and DBSCAN to the same data set which is in csv format, but i'm unable to get good results. I have attached the DBSCAN results.I,m getting overlapped clusters and can't figure out ...
- 11
0
votes
1
answer
39
views
Deep learning classification with multiple temporal data
I'm working on a project to predict the category of music segments in an audio file (represented in pianoroll format with an additional column for the corresponding class). Each row represents the ...
- 1
2
votes
0
answers
58
views
Understanding diagonal rescaling in multiplicative update rules for NMF
SUMMARY
How does the diagonal rescaling fit into the derivation of a multiplicative update rule for non-negative matrix factorization (NMF)?
DESCRIPTION
The NMF problem aims to find non-negative ...
- 21
0
votes
0
answers
85
views
Silhouette Score for ordered clusters
My clusters are arranged according to a time series, and I want to compute the silhouette score for the clustering performed, considering that they follow an order. Therefore the nearest cluster to ...
- 101
1
vote
0
answers
28
views
Learning based on correlations?
I am currently interested in a specific supervised learning sub-problem. We have access to Data X and targets y as in traditional statistical / supervised learning. However both X and y are very noisy....
- 1,665
1
vote
0
answers
41
views
Unsupervised Ranking
Assume we have N data points like
data = [(x1, y1, z1), (x2, y2, z2),....., (x_N, y_N, z_N)].
I am looking for unsupervised ranking algorithms. The thumb of rules ...
- 387
4
votes
1
answer
1k
views
How to compare labels from clustering analysis and original ones?
I was asked to run a clustering analysis to assess the validity of labels for a manually labelled dataset.
I can simply save the actual labels (4 classes: 0, 1, 2, 3) and run clustering analysis (let'...
- 127
1
vote
1
answer
261
views
Unsupervised learning: How to identify differences between clusters?
I'm learning about unsupervised learning and I tried to use KMeans, AgglomerativeClustering and DBSCAN on the same datase. The result was ok, they seems to work fine according silhouette_score() ...
