Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
482
questions
0
votes
0answers
8 views
Can Agglomerative Clustering (Heirarchical) form non-convex clusters?
I want to know whether Agglomerative Hierarchical clustering draws non-convex cluster boundaries. From sklearn's comparing diff clustering algo experiment it seems like Agglomerative clustering can ...
1
vote
0answers
11 views
Algorithms or techniques for pattern recognition [on hold]
I am looking for algorithms or techniques that could be used for pattern recognition/extraction in data. Preferably I would like to know if there are any algorithm implementations in Python that I ...
0
votes
1answer
30 views
In cluster analysis, is it better to normalize the data or standardize it?
In cluster analysis, is it better to normalize to $[0, 1]$ (i.e., $\frac{x-\min(x)}{\max(x)-\min(x)}$) the data or standardize via z-score (i.e., $\frac{x-\bar{x}}{s_x}$) it?
I know normalization ...
1
vote
0answers
18 views
One-Class SVM - Decision function
The following is based on the paper: Schölkopf et.al - SVM for Novelty Detection
First let us consider the (classical) Soft Margin SVM optimization problem:
${\displaystyle {\text{minimize }}{\frac {...
0
votes
0answers
22 views
Lead Qualification ML Algorithm Guidance
My company has tasked with me with building a ML model to qualify sales leads for our product, and after our brainstorming session, I am not sure how to approach the type of solution they seem to be ...
0
votes
0answers
7 views
Define Insurance Risk Score from Unsupervised Clustering + Supervised Classification using logits
I am trying to use a Neural Network to perform multiclass classification. The classes represent Insurance Risk Level. The most risky level is Level 1, the least risk corresponds to Level 10. The ...
0
votes
3answers
28 views
How to identify or give a meaning to the cluster membership in a hierarchical clustering?
I know clustering is a type of unsupervised learning problem, however when Kmean clustering is used one can sort the membership based on the cluster centers.
For example consider the cluster ...
1
vote
2answers
202 views
Machine learning - Normalize or Standarizing
When prepocessing continuous and integer data, is it better to normalize to $[0, 1]$ (i.e., $\frac{x-\min(x)}{\max(x)-\min(x)}$) or normalize via z-score (i.e., $\frac{x-\bar{x}}{s_x}$)? Is doing both ...
1
vote
0answers
7 views
Clustering of multiple very similar time series
so I have around 100 measurements of the current from a welding system. Each measurement has different sizes, but all are between 40k and 90k samples.
Some tests were done with very high (and ...
0
votes
0answers
21 views
How can I visualize and cluster weighted graphs in python?
On any ecommerce website, you have options to apply filters to filter out products. For example:
So I have data of how many users applied what filters tuples on the website. Which is fetched from ...
1
vote
0answers
14 views
How to fit mixture of gaussians with identical mean?
Say I have data generated by a mixture of gaussians whose components have the same mean, but very different covariances, like the one generated by this code:
...
0
votes
0answers
19 views
what does negative coherence score means?
I am working on topic modeling and when I tried to see the coherence of the topics all are negative.
Does anyone know what could be the interpretation behind negative score?
This is the example of ...
1
vote
3answers
44 views
Cluster categorical data (survey data)
I have a dataset containing around 800 observations:
It's a dataset collected via a survey; each row is a dataset filled with information re. diet habits, physical activity, the fact of taking ...
0
votes
0answers
13 views
When combining many algorithms, what are the techniques used to decide which algorithm is working ok and which one isn't?
There are some "technologies" like:
Elastic X-Pack
Darktrace
These use many unsupervised algorithms to find anomalies. As expected, algorithms do not agree on what are and what aren't anomalies. ...
0
votes
0answers
14 views
Unsupervised Clustering high dimentional data not having estimation for K
I have a dataset (all numerical) of 50K records containing 500 features. we are trying to find fingerprints. Meaning that we would like to cluster the data and report one of the nodes in each cluster ...
0
votes
1answer
10 views
Is learning label embedding by factorizing label co-occurrence matrix unsupervised learning?
I was working on creating embeddings for medical concepts. These terms/phrases are used for annotating biomedical documents. Now usually the method of creating a co-occurrence matrix and then ...
0
votes
1answer
85 views
K-means classifies 96% of my data in 1 cluster. Any suggestions to improve the results?
Problem: K-Means clustering shows 96% of my data belongs to one cluster. How can I improve my results or should I conclude that no cluster exists in my dataset. Dbscan clustering shows 1 cluster ...
0
votes
1answer
56 views
How to find similar users in a social network
I have a set of users from a social network. These users are represented by large sparse vectors. Let's say that a small subset of those users bought a ticket for a particular movie. How could I find ...
1
vote
1answer
14 views
Using k-means to segment customers in the positive class
I have some labeled data (0=didn’t cancel, 1=canceled) that I am creating a model for in my marketing class.
On top of predicting who is likely to cancel, I’d like to explore the possibility of ...
1
vote
0answers
19 views
Why mixture model with Gibbs sampling works?
I just have a question about why Gibbs sampling can correctly estimate parameters with random initial value?
That is to say,we can sample z by:
\begin{align}
p(z_i=k \,|\, \cdot)
&\...
0
votes
0answers
18 views
Interpreting association rules
I'm using mlxtend to find association rules in a dataset of products that customer bought over the span of a month. You can see an example of how it is used here.
The dataset has around 1000 types of ...
0
votes
0answers
48 views
mathematics behind autoencoders
Someone knows of an article that explains in detail the mathematics behind the auto-encoders.
All the articles I find show the typical diagrams
...
0
votes
1answer
41 views
Supervised anomaly detection of multiple time series
I'd like to develop a set of models for anomaly detection of multiple time series. After some reading, I have found a few promising approaches, such as
Segmentation-based approaches (SECODA);
...
0
votes
0answers
15 views
Unsupervised Learning vs. Supervised Learning in Natural Language Processing
I don’t quite understand the difference between supervised vs. unsupervised learning in Natural Language Processing, when trying to predict the next sequence of word in a sentence. Could someone ...
1
vote
0answers
36 views
calculate Inception Score and FID in GAN [closed]
Do we calculate Inception Score and Fréchet Inception Distance (FID) on the images generated by generator in parallel during training ? Or do we save the images generated by the generator and later ...
1
vote
0answers
31 views
Neural Network to discover an unknown number of patterns from a dataset of images?
I have a big set of images (>10.000), where there are similarities among them. I need to find a number/group of image patterns (eg, 5) that represent all images.
As I do not know what patterns are, ...
2
votes
1answer
99 views
LDA/NMF Topic Modeling vs Topic Modeling using “skip gram” approach
I am having a little friendly debate with my coworker on how to properly/optimally do topic modeling. I am just using the regular traditional nmf/lda approach and he decided to do it using "skip grams"...
0
votes
0answers
17 views
Help in Handling multiple classes in independent categorical variables and improve performance
The dataset has 4 categorical and 1 numerical variable and a timestamp variable. Out of 4, three categorical variables are having more than 100 categories.
I tried doing one-hot encoding on the whole ...
0
votes
1answer
23 views
Can NMF assign probabilities to the topics it outputs?
It's my understanding that only LDA can assign probabilities to words within each topic that it discovers since it's a probabilistic graphical model
politicians 0.05 united states 0.10 obama 0.20 ...
-1
votes
2answers
48 views
Similarity measure/metric for long time series
I'm looking for a similarity measure/metric to cluster long time-series datasets. I feel that Euclidean distance won't do any good for my application, for it is not robust enough to detect patterns ...
-1
votes
1answer
35 views
n for Hopkins statistics using get_clust_tendency
I started to use the Hopkins statistics to establish, if a dataset is 'clusterable'.
I am using the following code - taken from here:
...
1
vote
2answers
60 views
Running k-means clustering with k = 2 recursively on clusters greater than a certain size
Does it make sense to run k-means with k (number of clusters) of 2, and then for every cluster bigger than N, run k-means again with k = 2? We can then keep doing it until we have all clusters of size ...
0
votes
1answer
16 views
Can I use KernelPCA after using TruncatedSVD before clustering?
I am working on a project at a company where I have to make clustering/unsupervised model. The data I am working on is very sparse with high dimensions and after some research, I found out ...
0
votes
0answers
22 views
Extract data from mainly unstructured sets and derive risk metrics out of those
I have the following question (this was a real life problem):
Q: Extract data from mainly unstructured sets and derive risk metrics out of those. From what you know or imagine about the data ...
0
votes
2answers
25 views
Anomaly Detection Without a Baseline
I am attempting to find anomalies in accounting data (similar to this study: https://arxiv.org/pdf/1709.05254.pdf). I don't have any labeled data, so this attempt needs to be unsupervised. I am having ...
0
votes
0answers
14 views
Usage of VIF in unsupervised model
I'm working on building an unsupervised model for real time anomaly detection based on the concept of Randomized Matrix Sketching (http://www.vldb.org/pvldb/vol9/p192-huang.pdf) which involves ...
0
votes
1answer
26 views
Can I use the Silhouette to measure quality of clusters in different dimensions?
Can I use the Silhouette to measure quality of clusters in different dimensions?
For example, let's say we run kmeans for some $k$ using 6 features of the dataset. Mark the resulted silhouette as $...
0
votes
1answer
194 views
gensim LdaModel - How to reduce the number of words in each topic?
I'm trying to get more sparse topics (Less overlaps between different topics).
https://radimrehurek.com/gensim/models/ldamodel.html
I know it should be determined by the alpha parameter.
I've ...
0
votes
0answers
20 views
LDA: weight distributions of inferred documents
I have trained a two-topic Latent Dirichlet Allocation (LDA) model on a corpus and I am now inferring on a test corpus (the nature of the corpus is irrelevant). During inference, for each new document ...
0
votes
0answers
36 views
Unsupervised classification of images
Assuming I have a dataset of images from two similar classes, for example let's say 95% Labradors and 5% of Chihuahuas and I want to make a classifier. The point is that I need to find the anomalies (...
1
vote
0answers
48 views
Recognizing whether a written and spoken number is the same
For our ML assignment we have three datasets. The challenge is about checking whether a written and spoken number refer to the same number. We're using the MNIST dataset with handwritten numbers, and ...
1
vote
0answers
32 views
How to measure correlation between two groups of variables?
I have a data set that contain 75 variables of football players . These 75 variables basically measures two different types of information. 30 of those variables related to bio metric information ...
0
votes
0answers
12 views
Metric learning with respect to an outcome
Suppose I have $n$ datapoints in $p$-dimensional space, and the $p$ variables are highly heterogenous. That is, there is no natural way to combine them. Some are ordinal, some one-hot, some continuous,...
0
votes
0answers
25 views
Looking for ways to transform time-series data recorded from object movement into equation describing the movement direction of the object
Looking for some time-series data transformation advice!
I want to know what's the best way to transform data of 9-tuples time series data of IMU (Inertia Measurement Unit) sensor, recorded from a ...
1
vote
1answer
448 views
Deep learning models for unsupervised semantic segmentation
I am working on semantic segmentation for satellite images using keras and python. It is my understanding that popular models like U-Net require mask images (labels). Are there any unsupervised deep ...
1
vote
0answers
9 views
Is there a representer theorem for unsupervised learning (to justify kernel density estimation)?
In supervised learning, we get a representer theorem by considering regularized losses of the following form:
In Kernel Density Estimation, we simply directly assume densities of the form
Could this ...
1
vote
0answers
18 views
Predicting user behaviour based on transactional data - flagging “risky” behaviour
Firstly, I'm not sure if this is the right instance of StackOverflow to post on so feel free to ask me to put it elsewhere.
I am exploring the concepts of clustering and "unsupervised" learning for ...
0
votes
2answers
31 views
R - high dimension data using k means clustering [closed]
The dataset is 1000(observations) x 700(variables), After using pca to do dimension reduction, PC150 explained 85% Variance, so I use this (1000 x 150) data to do k means clustering.
This code was ...
1
vote
1answer
31 views
Hierarchical clustering for aggregrated features at higher thresholds/levels?
I am trying to use clustering on certain data. The data itself has three natural levels: at the lowest level the elements are fundamental building blocks, at the second level these fundamental ...
-1
votes
2answers
36 views
Unsupervised Clustering
My research is about comparing K-means and DBSCAN, and Im using unsupervised learning method in clustering.
Is it true that the number of cluster in K-means is also the same number as the unique ...
