Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
591
questions
0
votes
0answers
21 views
Techniques to measure similarities between instances and a dataset
I'd like to get a similarity measure between my dataset and a set of instances. For example, let's say I have a dataset with 4 features and n rows -
...
1
vote
1answer
75 views
unsupervised anomaly detection on sparse data
Given that I have a very sparse data matrix with continuous features, like this dataframe for example
...
1
vote
1answer
32 views
Confused between K-Means and Hierarchical Clustering for 9 different categories
I am trying to classify 9 different species of elephants into clusters using unsupervised learning. I have the following data about them:
Their height
Eye Colour
Sound they produce in decibel (dB)
I ...
0
votes
0answers
7 views
Is there a machine learning method for matching similar groupings?
I have a problem where I have rows/samples that are grouped together and each sample has a specific label (my data is genetic with genes being the samples and they are grouped together in the genome ...
0
votes
0answers
11 views
Is it viable to use ML models sequentially in a pipeline?
I have a machine learning model that predicts whether a gene is likely to cause a disease (the prediction is a probability score for a gene between 0 to 1, so a 1 score gene causes the disease and a 0 ...
0
votes
0answers
13 views
hierarchical clustering with categorical data
I have a dataset of 10K patients (row variables) with 10 disease conditions (such as heart condition, asthma, diabetes etc) along the columns. All the disease conditions are binary variables (yes/no).
...
0
votes
0answers
15 views
Outlier detection using 2D spatial information
I have a list of sensor measurements for air quality with geo-coordinates, and I would like to implement outlier detection. The list of sensors is relatively small (~50).
The air quality can gradually ...
0
votes
0answers
22 views
Extract Keyword/Concept From Column Description Using NLP
Suppose in my database, each table has a description associated with each column and I want to further extract keyword or key concept from the description. For example,
mean of transaction amount in ...
0
votes
0answers
7 views
How to determine a binary classifier threshold for unsupervised algorithms?
I am trying to create a model that can distinguish between normal and anomalous data. The training dataset contains non-anomalous data only. I am using an autoencoder that gets trained on this data to ...
2
votes
0answers
21 views
Is Normalized Mutual Info Score equivalent to V-measure when normalized by arithmetic mean
According to sklearn.metrics.v_measure_score, it says
This score is identical to normalized_mutual_info_score with the 'arithmetic' option for averaging.
In the ...
0
votes
0answers
10 views
General implications of low entropy for a dataset
I am fairly familiar with entropy, which quantifies uncertainty/surprisal of a random variable. In my case, I have a corpus where I can use empirical word frequencies to estimate entropy of the entire ...
1
vote
1answer
38 views
K means clustering breakup---galaxy spectrum data set
I have a spectrum data set (total 22000). Similar to an electronic wave data, two dimensional (Flux vs Wavelength). A typical set of wavelength plot looks like below
Now I am doing kmeans on this ...
0
votes
0answers
23 views
K-Means Clustering Problem in large 3D image CAD datasets
I have 1 million unclassified cad images in one folder and I need to cluster those images according to its similarity in 350 cluster folders, i.e k=350.
Out of 350 folders after clustering,
There are ...
0
votes
0answers
26 views
Clustering algorithm for a coordinate-based matrix
I have $1000$ scenarios, each of which is composed of $5$ users' coordinates $(x_i,y_i), \forall i \in \{1,\dots,5\}$. Now, based on users' coordinates, I want to cluster these $1000$ scenarios into ...
0
votes
0answers
6 views
Explanation of Excess Mass(EM)
I was researching on evaluation metrics to understand the performance of unsupervised anomaly detection algorithms and I came across this paper
The author suggests that EM and MV based numerical ...
0
votes
0answers
9 views
Clustering DBSCAN's parameter Epsilon: How is eps related to scale of data being clustered?
How is scale of eps related to data to be clustered in DBSCAN?
e.g. in image of 1024x1024, we have points as:
...
0
votes
0answers
11 views
Best way to suggest 'n' number of users based on demographic, interests data
I have to build a recommendation engine which suggests 'n' number of similar users to a user.
I tried to implement this in user-user recommendation system methodology and unsupervised learning.
Theres ...
1
vote
3answers
51 views
Unsupervised Anomaly Detection with groups
Let's say we are a bank and are interested in catching fraudulent customers. We gather ~100.000 independent samples of 40 independent variables and 4 are behavioral variables (what a customer does).
...
1
vote
0answers
12 views
autoencoders for radiographs - do watermarks affect the performance considerably
I have to implement an autoencoder to reconstruct the input radiographs and do unsupervised feature learning in the process. However, the radiographs that I have contain some watermarks like X-ray ...
2
votes
0answers
18 views
How can I cluster sequential data?
Suppose that I have a sequence of vectors $y_n \in \mathbb{R}^m$ for $n \in \{1, \dots, N\}$. My goal is to divide $y_n$ in $K$ clusters and want my clusters to satisfy the following conditions:
Each ...
1
vote
1answer
40 views
is k-means generalizable at any distance? [duplicate]
The classical version of k-means uses the Euclidean distance in the first step, and the arithmetic mean (the value center) in the second step. Is k-means generalizable to other distances and other ...
0
votes
1answer
14 views
Why doesn't t-SNE need the labels before visualization?
In the original Maaten and Hinton paper, they explicitly say that the class membership is not used by the t-SNE calculations, only for picking colors in the plot.
For all of the data sets, there is ...
0
votes
0answers
7 views
What is the statistical relevance of gamma in k-prototypes algorithm and why is it related to the standard deviation of the numeric columns?
The k-prototype algorithm uses gamma to provide weight to the categorical features. I have a few queries regarding it :
Why is there no upper limit to it? Should it not be (1-gamma) such that gamma ...
0
votes
0answers
19 views
Fixing the parameters of the variational distribution in Expectation-Maximization
Consider directed graphical model $z \to x$ (with $z$ unobserved and $x$ observed). The evidence lower bound on the log-likelihood $\log p(x) = \log \sum_z p(x, z; \theta)$ for parameters $\theta$ (...
1
vote
0answers
9 views
Best practice/Ideas for clustering Event Sequence Embeddings?
My dataset consist of around 40 000 samples of event sequences.
Sample of data
[[Event 1, Event 2, Event 4, Event 5],
[Event 1, Event 3, Event 4],
[...]]
I ...
1
vote
0answers
19 views
What is the meaning of noise in a dataset with no dependant variable?
My understanding of noise & signal comes from the context of bias-variance tradeoff in supervised methods.
But given a dataset with no dependant variable, how do you define noise? & how do you ...
1
vote
0answers
11 views
Updating an unsupervised model but retaining similarity
In my example I am using topic modelling (specifically a version of LDA) although I think avenues for exploring this could relate to other unsupervised techniques like clustering.
I train a model and ...
1
vote
2answers
38 views
How can PCA maximise variance after I standardise all predictor variance = 1?
I have been reading about Principal Components Analysis, and I think it is in general trying to extract as much "variance" out of the predictors $ \vec{X} = (X_1, X_2, ..., X_n)$ by ...
0
votes
0answers
17 views
Latest research and explanation on how semi-supervised learning is performing better than supervised?
So in AAAI 2020 also semi-supervised learning is given the push. There are some intuitive reasoning provided by people but since the research is so fast, I wanted to know actually what is the latest ...
0
votes
0answers
19 views
Evaluation method for outlier detection with a data that has no labels
I am applying an unsupervised outlier detection model called isolation-Forest to detect outliers using unlabelled time-series data. I do not have labels that distinguish a true outlier from a false ...
0
votes
0answers
9 views
Clustering Algorithms for News Aggregator
I'm working on a news aggregator, and I am trying to get very nice clusters for similar articles. I'm only working with a few hundred articles so speed is not a concern for me. I already have plans as ...
0
votes
0answers
17 views
Compare similarity/dissimilarity b/w 2 clustering results from different datasets
So I am trying to find out how similar are 2 clustering results (hierarchial). 1st dataset comprises of 10 features. 2nd dataset comprises of these 10 features plus 5 additional features. So my basic ...
0
votes
1answer
32 views
Auto-encoders' learning process and overfitting
Reading about autoencoders from Ian Goodfellow's deep learning book, and they made this statement about autoencoders learning process on page 494:
"Unfortunately, if the encoder and the decoder ...
1
vote
1answer
28 views
Definition of boundary points in DBSCAN
If I understand correctly, DBSCAN is the following method of decomposing a set, which has parameters $\epsilon$ and $k$. Given a set $E$ of points in $\mathbb R^n$, consider the graph $G$ whose ...
9
votes
4answers
2k views
Are all Machine Learning algorithms divided into Classification and Regression, not just supervised learning?
I'm newbie in AI
I know that Supervised Learning algorithms are divided into Classification and Regression algorithms.
But is that true of all machine learning algorithms, not just Supervised Learning?...
0
votes
0answers
8 views
Reverse engineer unsupervised algo using supervised one
Let's say that through some unknown process, each row in a dataset is labelled with an integer between 1 to 10 inclusive. Now, if I run random forest for example, and get a promising result for ...
2
votes
1answer
54 views
Can a Gaussian Mixture model be fit with a continuous response variable?
Does the Gaussian Mixture model require binary and multiclass response/target variable (classification), or can the target vector consist of all real numbers (continuous variable, regression)? Why is ...
1
vote
0answers
7 views
Prerequisites/Checks for performing clustering
What are the checks that should be done on our data before performing clustering? Like how to check whether the dataset contains clusters of equal size/density or the clusters present in the dataset ...
3
votes
0answers
41 views
When to use K-Medoids instead of K-means
When it's better to use K-Medoids rather than K-Means? Can anybody give some examples of dataset for the same?
1
vote
1answer
117 views
Limitations of K-Means Clustering [duplicate]
I was going through a document of Western Michigan University to understand the limitations of K-means clustering algorithms. Below is the link:
https://cs.wmich.edu/alfuqaha/summer14/cs6530/lectures/...
1
vote
0answers
20 views
How do you train a clustering model?
This should've been a pretty simple question, but I still have a few questions so I decided to bring the discussion here.
The thing is, I have a group of products, and the historical dataset looks ...
4
votes
1answer
80 views
Is Anomaly Detection Supervised or Un-supervised?
AFAIK - One way to process data faster and more efficiently is to detect abnormal events, changes, or shifts in datasets. Anomaly detection, also known as outlier detection is the process of ...
0
votes
2answers
97 views
Feature extraction in deep neural networks
From many definitions that I read, I concluded that a DNN (deep neural network) is an ANN (artificial neural network) that have more than one hidden layer.
Knowing that CNN (convolutional neural ...
0
votes
0answers
13 views
Gaussian Mixture Model Clustering - cluster means are assigned to a different cluster
I ran a gaussian mixture model with 7 clusters on my data. My data has been PCA transformed with 200 components. Then I extracted the means of each cluster and applied the predict_proba function on ...
2
votes
1answer
22 views
What statistical tests or machine learning techniques would be best for discrete variables with many levels?
I have a dataframe with 20 categorical variables, each with 30+ levels. As a result I don't have a target variable on hand per-say but I would like to use statistical techniques or machine learning to ...
0
votes
0answers
23 views
Why is my training unstable?
I am training a Variational autoencoder with and without data labels. When I use labels (blue line), validation error decreases with epochs but without labels (orange line) the training is unstable. ...
0
votes
1answer
32 views
Does automatic feature selection for clustering helps with finding meaningful clusters?
The objective of clustering is to find interesting groups in data. My question is, whether feature selection can substantially help with this objective.
I understand feature selection can remove ...
1
vote
0answers
29 views
How to achieve good clustering results in subsequence time series clustering with DBSCAN?
I want to find patterns in a time series and use clustering for that. Before I cluster, I create subsequences from the time series using a sliding window approach. (STS-clustering)
So far I have tried ...
1
vote
1answer
16 views
Differentiability in Generative Adversarial Networks
I've got some questions about the differentiability condition of GAN's, i.e. both G and D need to be differentiable wrt. their inputs and the parameters describing them. It's of more mathematical ...
0
votes
0answers
25 views
stepwise clustering
Assume that we have 3 features in our dataset and we aim to cluster them. Assume that first two variables are in the same scale and have a "similar nature" and the third one has totally different ...
