Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
616
questions
1
vote
0answers
8 views
How to compare the peformance of different clusterings without true labels
Firstly, I know some scores like silhouette score and Davies–Bouldin score to compare the performance in one clustering method. However, I am not sure how to compare results in different clustering ...
0
votes
0answers
6 views
Is there a clustering algorithm that can divide population for few classes and neglect some samples?
As far as I know all clustering algorithms assume that all delivered data points have to find its cluster.
Is there an algorithm that could focus only on n clusters (number stated by user) and try to ...
0
votes
0answers
10 views
HDBSCAN Soft clustering - Why everything becomes 1 class?
GOAL
Using HDBSCAN
Putting all data points in to a class and leaving behind 0 point as noise (in other words partitioning the data without leaving behind any unallocated data points)
DONE
I have a ...
0
votes
0answers
24 views
Difference between Self-Taught learning and Transfer Learning?
I am new to Transfer Learning and I start by reading A Survey on Transfer Learning, and it stated the following:
according to different situations of labeled and unlabeled data in the source domain, ...
0
votes
0answers
21 views
Feature engineering for fraud detection with Isolation Forest
I am researching and doing a project related to the detection of fraudulent transactions in the financial system. For this research we are working with unsupervised learning, more precisely we are ...
2
votes
2answers
22 views
Does the mean normalization step of k-means affect its performance?
A common step when clustering using k-means is to first standardize the dataset so that each feature has zero mean and unit variance.
I understand why forcing unit variance helps k-means generate ...
0
votes
0answers
23 views
How to group every data point with HDBSCAN to some group to have no noise?
TASK
I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits)
I use HDBSCAN to do it
GOAL
The goal is when users come on our site and I can ...
0
votes
0answers
23 views
Cluster 1D data with noise tolerance
Let's say I have a 1D list :
[0,0, 1,1,1, -1,-1,-1,-1, 1,1,-1,-1,1,1,1,0,1,1,0,-1,-1,1,-1]
I wish to separate them into groups of -1 and 1, with certain tolerance ...
0
votes
0answers
7 views
How to use Product Matching to create Product Bundles
I am working on a product matching model.
GOAL
A store has many products like creams, perfumes, other beauty products.
Based on product properties I have to cerate bundles of it so we can sell more ...
0
votes
0answers
6 views
Implementing nested features in unsupervised models
Our project has built an unsupervised model that uses data about a number of companies. Some of these companies are public and some are private. The ones that are public have much higher financial ...
1
vote
0answers
21 views
Why limit the volume of the autoencoder in self supervised learning?
In this article: self-supervised learning: The dark matter of intelligence the authors tried to unify the self-supervised learning for tasks with discrete outcomes and continuous outcomes. They ...
0
votes
0answers
13 views
Preprocessing on a “branch-like” dataset of varying density
I'm trying to classify a dataset with unsupervised learning. Based on a limited amount of hand-labelled data, I was able to identify two larger sets of points that probably belong to the same class, ...
1
vote
0answers
23 views
outlier detection for sparse data in categorical variable
I have a big dataset with a column "clientid" and a categorical column "choice".
I want to find out what are the clients that have strange combinations of choices (less frequent ...
0
votes
0answers
25 views
Non-clustering Unsupervised Learning Examples
in one of my classes I was asked to explain the difference between unsupervised and supervised learning. I said that unsupervised learning uses tricks about the properties of statistical ...
0
votes
0answers
24 views
'False Positive' detection in corona testing using ML techniques
I am working on the detection of false positive test result in COVID-19 testing. The false positive sample are mainly due to the contamination during the routine testing. Samples are handled in batch ...
0
votes
0answers
10 views
How to find the outliers(trending data) for real estate data
I have prices, location, type, rooms, etc info about various houses from a real estate website. I want to find the trending data or the houses which are unique, e.g. a house in New York, with 2 rooms ...
0
votes
0answers
14 views
Can I use Customer ID or count(customer ID) in K means clustering? Also, how to find the data points that doesn't fit the clusters well(outliers)?
I am trying to cluster a large dataset with 150k observations. There are 4 variables: Customer ID, Type(credit/Debit), Amount and Country(4 levels). Usually, we don't use customer ID as it doesn't ...
0
votes
0answers
7 views
How good is zero-shot learning for few-shot/low-shot learning?
I currently face a few-shot/low-shot learning problem in NLP domain. This problem is just like the definition of the problem itself: some labels appears very infrequently.
Based on my experiments, ...
0
votes
0answers
22 views
Silhouette Coefficient acceptable value
Does anybody know about the acceptable values for Silhouette Coefficient (or maybe Calinski-Harabasz and Davies-Bouldin index) in K-means clustering?. I know that Silhouette Coefficients close to 1 ...
1
vote
1answer
46 views
Cubic Clustering Criterion in Python
Does anybody know if any package calculates the Cubic Clustering Criterion (CCC) index and the Approximate Expected R Square (http://documentation.sas.com/?docsetId=emref&docsetTarget=...
1
vote
1answer
21 views
In Latent Dirichlet allocation, is the following formula the probability of observing a single document, or an entire corpus?
This is the formula in question:
Source: https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
1
vote
1answer
33 views
Determine if a cluster has to be divided to 2 mini-clusters
I have a dataset containing clusters of points (x,y). Those clusters are given and I can't change them.
Some of those clusters have 2 "mini clusters" in them - so I need to do some ...
0
votes
1answer
12 views
Using an Autoencoder for Unsupervised Outlier Detection - Train and Predict on Same Data?
I have a high-dimensional dataset of medical utilization for a public health plan's membership in which I would like to identify outliers. i.e., which individuals are potentially over- and/or under-...
0
votes
0answers
8 views
What has metric learning achieved and what is the most prominent example in this field?
Can someone please educate an ill-informed person like me as to what the field of "metric learning" has achieved, e.g., a prominent example, and what are some of the most successful/...
1
vote
0answers
9 views
On quantifying the amount of information per example provided to the model in Supervised vs Self-supervised learning
I've seen Yann Lecun in his self-supervised learning talks talking about how traditional supervised learning (Classification setting) by attributing a class out of N classes to each example the ...
2
votes
1answer
46 views
Why are language modeling pre-training objectives considered unsupervised?
Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
0
votes
0answers
12 views
How can I smooth the predictions of a supervised model by learning its trend?
This is the predictions of a binary classification model. The model is doing predicitons continuously, and these values are the sum of positive labels during a 10 hours period. As you can see, some of ...
0
votes
0answers
5 views
Unsupervised training a Neural network to identify/estimate optimal threshold (scalar) between 2 clusters/distributions
I want to train a neural network to identify "optimal" threshold value which Separates between 2 clusters/distributions given a data set or a histogram.
For examle, say I have a 1-...
0
votes
0answers
25 views
Why can the L1 Norm be expressed as a constrain?
I am learning why the L1 regulariser (Lasso) is used to encourage sparsity in ML models. When describing the proof, I am seeing that the regularised minimisation cost function;
$$
min(RSS(w) + \lambda*...
0
votes
0answers
38 views
obtaining proximity matrix from random forest for unsupervised scenario in R
I recently came across the concept of proximity matrix in random forests (see for example this great StatQuest video). This can easily be obtained in the regression or classification scenario like so:
...
0
votes
0answers
23 views
Gibbs Sampling in LDA - any other inference procedures?
I am playing with the LDA model in python and would like to ask if anyone knows whether Python's toolkits for LDA (gensim/scikit-learn), have all implemented the Gibbs Sampling inside or is there any ...
0
votes
1answer
48 views
PCA: axis of least inertia or of greatest variance
In this video, it is said that:
The axis of least inertia [is the axis] of greatest variance.
However, this link says the axis of least inertia is also the axis of ...
0
votes
0answers
9 views
How to choose the best model for Non Negative Matrix Factorization?
I am applying NMF with NMF R package. In the early stages, I'm comparing three algorithms (Lee, Brunet,nsNMF) visualizing how fast they converge and how much they reduce residues as in the image down ...
1
vote
0answers
12 views
Using additional features while indexing in Approximate Nearest Neighbors
I am trying to develop a recommender system that suggests top 10 workers suitable for a task at hand. For the features, I have work type, criticality and location of work in both my historical and ...
0
votes
1answer
43 views
What is the-state-of-art for unsupervised Anomaly models through unlabeled data regarding evaluation/validation in 2020?
I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Since outlier detection is commonly considered ...
0
votes
0answers
8 views
Large unsupervised problem (clustering). Where do I start?
I’m working on a dataset that is about 5,000,000 observations with about 500 variable. My goal is to find “some interesting things” about the data. I,e correlation between variables, similarities, ...
0
votes
0answers
15 views
AutoEncoder and Unsupervised Clustering
I am working on a dataset of ~300 samples with ~5000 data-points each - ranged between 0 and 100. I am interested in:
Group samples for similarity;
Find the differences between groups;
Would make ...
1
vote
1answer
111 views
Is it possible to find cluster centroids in kernel K means?
Suppose ${x_1, \ldots, x_N}$ are the data points and we have to find $K$ clusters using Kernel K Means.
Let the kernel be $Ker$ (not to confuse with $K$ number of clusters)
Let $\phi$ be the implicit ...
0
votes
2answers
155 views
Hierarchical clustering and Dendrogram interpretation
I'm quite new to cluster analysis and I was trying to perform a hierarchical clustering algorithm (in R) on my data to spot some groups in my dataset. Initially, I tried with the k-means, with the ...
0
votes
0answers
16 views
Blind source separation using FastICA
Reading the example by scikit-learn on how to use the FastICA function, I couldn't understand the following plot:
In the third plot ("ICA recovered signal") - why is the magnitude different ...
0
votes
0answers
42 views
How to cluster similar images with tensorflow?
I have a set of photographs and I would like to cluster based on their similarity in order to identify the ones that look alike and eventually select the best one among each cluster. I would like to ...
1
vote
0answers
20 views
How does the coloring of dendrograms in SciPy work?
So I am clustering my data using linkage extensions. When I plot the diagrams of the dendrograms scipy chooses to color branches in different colors according to a "color threshold". As I ...
0
votes
0answers
9 views
Technique for clustering bivariate data with no-overlap on one variable?
I am looking for a technique of clustering bivariate data. Basically, I have X and Y and I want to divide the data into k groups. However, on one variable, let's say Y, there can be no overlap of ...
1
vote
0answers
37 views
How to learn a neural network based on a cost function in an unsupervised way?
I need to learn a neural network that predicts L (see equation below). So based on three points from a discrete trajectory I want to learn L by minimizing the equation below, which is a physical ...
2
votes
1answer
303 views
KMeans clustering - can inertia increase with number of clusters
I am doing kmeans clusters on sales data and i see that inertia increases for the initial increase in the number of clusters. Can you please explain why that happens?
I am doing Batched Kmeans for the ...
0
votes
0answers
28 views
Techniques to measure similarities between instances and a dataset
I'd like to get a similarity measure between my dataset and a set of instances. For example, let's say I have a dataset with 4 features and n rows -
...
1
vote
1answer
112 views
unsupervised anomaly detection on sparse data
Given that I have a very sparse data matrix with continuous features, like this dataframe for example
...
1
vote
1answer
46 views
Confused between K-Means and Hierarchical Clustering for 9 different categories
I am trying to classify 9 different species of elephants into clusters using unsupervised learning. I have the following data about them:
Their height
Eye Colour
Sound they produce in decibel (dB)
I ...
0
votes
0answers
8 views
Is there a machine learning method for matching similar groupings?
I have a problem where I have rows/samples that are grouped together and each sample has a specific label (my data is genetic with genes being the samples and they are grouped together in the genome ...
0
votes
0answers
12 views
Is it viable to use ML models sequentially in a pipeline?
I have a machine learning model that predicts whether a gene is likely to cause a disease (the prediction is a probability score for a gene between 0 to 1, so a 1 score gene causes the disease and a 0 ...
