Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
629
questions
0
votes
0answers
13 views
Bias-Variance tradeoff with Clustering algorithms
I'm investigating the bias-variance tradeoff in well-known machine learning algorithms. However, I'm not sure this concept applies in the case of unsupervised methods such as clustering algorithms.
Is ...
0
votes
0answers
13 views
Metric for comparing supervised and unsupervised model
I'm searching on how to compare (validate) a supervised learning model to an unsupervised one.
Let's say I have a supervised model for fault diagnosis which can tell me how accurate it is to predict ...
0
votes
0answers
8 views
Performance differences between variational autoencoder and t-SNE
I trained a convolutional variational autoencoder on a dataset of medical images to detect anomalies. Based on the reconstruction error, I try to distinguish between normal images and anomalies. In ...
4
votes
3answers
900 views
How do you learn labels with unsupervised learning?
In https://huyenchip.com/machine-learning-systems-design/design-a-machine-learning-system.html#design-a-machine-learning-system-dwGQI5R, I came across the sentence:
Similarly, you can use ...
0
votes
0answers
18 views
Unsupervised Clustering of large multi-dimentional data
Hello I am a machine learning newbie. I need some help with unsupervised clustering of high dimentional data. I have data with over 15 dimensions with around 50 - 80 thousand rows. The data looks ...
0
votes
1answer
12 views
When using "silhouette coefficient" to evaluate an unsupervised model, do we need a labelled dataset?
How does "silhouette coefficient" can find the optimal number of clusters when the dataset is not labelled? Does it need a labelled dataset or is it pure statistics?
I mean, when doing ...
0
votes
0answers
20 views
Clustering machine learning algorithms
I am looking for a clustering algorithm that can separate my data into clusters. The data has three different variables and there are approx 500k samples. I need some control over the number of ...
1
vote
0answers
11 views
select the final clustering based on agreement of trials between two groups
There are well-recognized cross-validation based methods to select the number of clusters (e.g. in this answer) . However, suppose I know the number of clusters beforehand, can I select the final ...
0
votes
0answers
19 views
Unsupervised feature selection on mixed data
Is there any unsupervised feature selection possibility (for mixed data) ideally in Python? I have all kinds of data types in my dataset (scale, ordinal, binary, cathegorical). Is there any option how ...
0
votes
1answer
30 views
Smart Statistics vs Unsupervised Learning for Anomaly Detection
I'm a working on an Anomaly Detection project for college. Please do help me understand some concepts.
Some of my biggest questions are:
How do we decide for a project on Anomaly Detection, should we ...
0
votes
0answers
22 views
What is the prior probability in a Dynamic Naive Bayes classifier?
For a Hidden Markov process with multiple types of emissions, it is possible to perform current state classification using the Naive Bayes likelihood estimation:
$ p(j|b,d) \propto p(b|j) \cdot p(d|j) ...
1
vote
0answers
44 views
How to detect anomalies in multiple different IP addresses?
Given that my input data consists of various destination IP addresses and its incoming connections from source IP addresses with country codes during certain timestamps, I would like to detect ...
2
votes
0answers
13 views
Clustering data with temporal effect
As an example of a dataframe, we could think a problem such as:
(my real table is much bigger and with a lot of ID_PRODUCTs and a lot of ID_CLASS_PRODUCTs)
ID_CLIENT
DATE
ID_PRODUCT
ID_CLASS_PRODUCT
...
2
votes
0answers
21 views
Event detection in multivariate time series
I've have the following multivariate one second time series composed of that I can manually label and its based on sudden value changes (increase)from zone 1 to zone 2 and (decrease) from zone 2 to 3.
...
1
vote
1answer
30 views
Terminal State Classification with Hidden Markov Models
The Viterbi algorithm predicts the most likely sequence of hidden states. But what if the variable of interest is the final hidden state? For example, predicting if a friend (whom you can't visit due ...
2
votes
0answers
18 views
How to extract simple shapes from a feature map?
I am working on image parsing project. I want to find a way to automatically parse an object into a list or a graph of simpler shapes. Is there any practical information on how to do so?
So far I took ...
2
votes
0answers
100 views
Marginal distributions of the Indian Buffet Process
In the construction of the Indian Buffet Process we have that customer $n_1$ chooses $\mbox{Poisson}\left(\frac{\alpha}{1}\right)$ dishes, $n_2$ chooses $\mbox{Poisson}\left(\frac{\alpha}{2}\right) \...
1
vote
1answer
43 views
How to compare clustering results between "raw" and normalized data
I have a dataset and I would like to apply a clustering algorithm to find some groups. I do not have any label, so it is just wondering if I can find relevant clusters. If it may help, it is ...
0
votes
0answers
22 views
Predictions from Robust PCA
I'm interested in Robust Principal Components, which are implemented in R package rpca.
As per the definition, the method constructs two matrices $S$ and $L$, ...
0
votes
0answers
16 views
Principal Component Analysis that Minimizes Alternative Error Metric
The standard Principal Component Analysis (PCA) method minimizes the so-called reconstruction error written as $\sum_k||\hat{\mathbf{x}}^k-\mathbf{x}^k||^2$, where the elements $\hat{\mathbf{x}}^k$ ...
1
vote
0answers
14 views
Does scikit-learn implement the SLINK algorithm for single-linkage, hierarchical, agglomerative clustering?
Does scikit-learn implement SLINK for single-linkage, hierarchical, agglomerative clustering? I wasn't able to find that info in scikit-learn's documentation.
0
votes
0answers
28 views
Does this problem requires Supervised Learning or Unsupervised Learning
I have 50 Features in a Dataset to predict 1 Variable "Units Sold". I am currently using XGBoost model (Supervised Learning) to train all these 50 Features and the accuracy of the model on ...
1
vote
0answers
9 views
Aggregating results from multiple unsupervised clustering studies
Is there a framework, similar to meta-analysis, to interpret or aggregate the results from multiple unsupervised clustering studies?
For example, take two studies on evaluating patients with heart ...
0
votes
0answers
31 views
Are word2vec, contrastive predictive coding, etc. examples of "energy based models"?
I am trying to place contrastive learning models, such as word2vec and Contrastive Predictive Coding [1] in the context of other generative models, such as Autoregressive Models, VAEs, GANs, and ...
0
votes
0answers
25 views
Anomaly Detection in Highly Variable Time-Series Data
I am trying to detect anomalies through a column called count. The data is a time-series data and it is present for every 5 minutes for each day. The dataframe looks like this:
...
1
vote
0answers
29 views
Avoiding overfitting in unsupervised ML
I am using a unsupervised pattern matching approach to create a trade strategy. I use the output of the pattern matched results to decide whether to enter a trade or not. For deciding the best pattern ...
0
votes
0answers
9 views
Social network data and Clustering analysis
I'm working with Twitter data, and I would like to separate Twitter users into different groups (or cliques) based on their "friendship" relations.
In particular, I want to do something like ...
0
votes
0answers
24 views
What the difference between clustering methods?
I am focusing on a thesis for introducing clustering methods. Chapter 3 includes hierarchical clustering, partitional clustering and density-based clustering. Meanwhile, chapter 4 is mainly on Self-...
0
votes
0answers
4 views
Classifying debtors with unsupervised-learning
I have unlabelled dataset which contains financial information of debtors. My goal is to determine the characteristics that tell whether a debtor is likely to pay the debt up, or not.
The dataset has ...
1
vote
1answer
38 views
Does the Membership Matrix of Fuzzy C-Means Clustering contain probabilities or degrees of membership?
I recently heard a lecture on Fuzzy C-Means Clustering that stated that the Membership Matrix contains probabilities that particular data points are members of particular clusters. I was confused by ...
1
vote
0answers
19 views
How to compare the peformance of different clusterings without true labels
Firstly, I know some scores like silhouette score and Davies–Bouldin score to compare the performance in one clustering method. However, I am not sure how to compare results in different clustering ...
0
votes
0answers
13 views
Is there a clustering algorithm that can divide population for few classes and neglect some samples?
As far as I know all clustering algorithms assume that all delivered data points have to find its cluster.
Is there an algorithm that could focus only on n clusters (number stated by user) and try to ...
0
votes
0answers
74 views
HDBSCAN Soft clustering - Why everything becomes 1 class?
GOAL
Using HDBSCAN
Putting all data points in to a class and leaving behind 0 point as noise (in other words partitioning the data without leaving behind any unallocated data points)
DONE
I have a ...
0
votes
0answers
167 views
Feature engineering for fraud detection with Isolation Forest
I am researching and doing a project related to the detection of fraudulent transactions in the financial system. For this research we are working with unsupervised learning, more precisely we are ...
2
votes
2answers
30 views
Does the mean normalization step of k-means affect its performance?
A common step when clustering using k-means is to first standardize the dataset so that each feature has zero mean and unit variance.
I understand why forcing unit variance helps k-means generate ...
0
votes
0answers
53 views
How to group every data point with HDBSCAN to some group to have no noise?
TASK
I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits)
I use HDBSCAN to do it
GOAL
The goal is when users come on our site and I can ...
0
votes
0answers
24 views
Cluster 1D data with noise tolerance
Let's say I have a 1D list :
[0,0, 1,1,1, -1,-1,-1,-1, 1,1,-1,-1,1,1,1,0,1,1,0,-1,-1,1,-1]
I wish to separate them into groups of -1 and 1, with certain tolerance ...
0
votes
0answers
10 views
How to use Product Matching to create Product Bundles
I am working on a product matching model.
GOAL
A store has many products like creams, perfumes, other beauty products.
Based on product properties I have to cerate bundles of it so we can sell more ...
0
votes
0answers
6 views
Implementing nested features in unsupervised models
Our project has built an unsupervised model that uses data about a number of companies. Some of these companies are public and some are private. The ones that are public have much higher financial ...
1
vote
0answers
23 views
Why limit the volume of the autoencoder in self supervised learning?
In this article: self-supervised learning: The dark matter of intelligence the authors tried to unify the self-supervised learning for tasks with discrete outcomes and continuous outcomes. They ...
0
votes
0answers
13 views
Preprocessing on a "branch-like" dataset of varying density
I'm trying to classify a dataset with unsupervised learning. Based on a limited amount of hand-labelled data, I was able to identify two larger sets of points that probably belong to the same class, ...
1
vote
0answers
34 views
outlier detection for sparse data in categorical variable
I have a big dataset with a column "clientid" and a categorical column "choice".
I want to find out what are the clients that have strange combinations of choices (less frequent ...
0
votes
0answers
31 views
Non-clustering Unsupervised Learning Examples
in one of my classes I was asked to explain the difference between unsupervised and supervised learning. I said that unsupervised learning uses tricks about the properties of statistical ...
0
votes
0answers
34 views
'False Positive' detection in corona testing using ML techniques
I am working on the detection of false positive test result in COVID-19 testing. The false positive sample are mainly due to the contamination during the routine testing. Samples are handled in batch ...
0
votes
0answers
19 views
How to find the outliers(trending data) for real estate data
I have prices, location, type, rooms, etc info about various houses from a real estate website. I want to find the trending data or the houses which are unique, e.g. a house in New York, with 2 rooms ...
0
votes
0answers
43 views
Can I use Customer ID or count(customer ID) in K means clustering? Also, how to find the data points that doesn't fit the clusters well(outliers)?
I am trying to cluster a large dataset with 150k observations. There are 4 variables: Customer ID, Type(credit/Debit), Amount and Country(4 levels). Usually, we don't use customer ID as it doesn't ...
0
votes
0answers
11 views
How good is zero-shot learning for few-shot/low-shot learning?
I currently face a few-shot/low-shot learning problem in NLP domain. This problem is just like the definition of the problem itself: some labels appears very infrequently.
Based on my experiments, ...
1
vote
1answer
25 views
In Latent Dirichlet allocation, is the following formula the probability of observing a single document, or an entire corpus?
This is the formula in question:
Source: https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
1
vote
1answer
36 views
Determine if a cluster has to be divided to 2 mini-clusters
I have a dataset containing clusters of points (x,y). Those clusters are given and I can't change them.
Some of those clusters have 2 "mini clusters" in them - so I need to do some ...
0
votes
1answer
18 views
Using an Autoencoder for Unsupervised Outlier Detection - Train and Predict on Same Data?
I have a high-dimensional dataset of medical utilization for a public health plan's membership in which I would like to identify outliers. i.e., which individuals are potentially over- and/or under-...
