Questions tagged [unsupervised-learning]
Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.
604
questions
1
vote
0answers
5 views
On quantifying the amount of information per example provided to the model in Supervised vs Self-supervised learning
I've seen Yann Lecun in his self-supervised learning talks talking about how traditional supervised learning (Classification setting) by attributing a class out of N classes to each example the ...
1
vote
1answer
13 views
Why are language modeling pre-training objectives considered unsupervised?
Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
0
votes
0answers
9 views
How can I smooth the predictions of a supervised model by learning its trend?
This is the predictions of a binary classification model. The model is doing predicitons continuously, and these values are the sum of positive labels during a 10 hours period. As you can see, some of ...
0
votes
0answers
5 views
Unsupervised training a Neural network to identify/estimate optimal threshold (scalar) between 2 clusters/distributions
I want to train a neural network to identify "optimal" threshold value which Separates between 2 clusters/distributions given a data set or a histogram.
For examle, say I have a 1-...
0
votes
0answers
24 views
Why can the L1 Norm be expressed as a constrain?
I am learning why the L1 regulariser (Lasso) is used to encourage sparsity in ML models. When describing the proof, I am seeing that the regularised minimisation cost function;
$$
min(RSS(w) + \lambda*...
0
votes
0answers
17 views
obtaining proximity matrix from random forest for unsupervised scenario in R
I recently came across the concept of proximity matrix in random forests (see for example this great StatQuest video). This can easily be obtained in the regression or classification scenario like so:
...
0
votes
0answers
16 views
Gibbs Sampling in LDA - any other inference procedures?
I am playing with the LDA model in python and would like to ask if anyone knows whether Python's toolkits for LDA (gensim/scikit-learn), have all implemented the Gibbs Sampling inside or is there any ...
0
votes
1answer
37 views
PCA: axis of least inertia or of greatest variance
In this video, it is said that:
The axis of least inertia [is the axis] of greatest variance.
However, this link says the axis of least inertia is also the axis of ...
0
votes
0answers
8 views
How to choose the best model for Non Negative Matrix Factorization?
I am applying NMF with NMF R package. In the early stages, I'm comparing three algorithms (Lee, Brunet,nsNMF) visualizing how fast they converge and how much they reduce residues as in the image down ...
1
vote
0answers
12 views
Using additional features while indexing in Approximate Nearest Neighbors
I am trying to develop a recommender system that suggests top 10 workers suitable for a task at hand. For the features, I have work type, criticality and location of work in both my historical and ...
0
votes
1answer
27 views
What is the-state-of-art for unsupervised Anomaly models through unlabeled data regarding evaluation/validation in 2020?
I'm researching anomaly detection, which is nothing else than outliers detection on a set of time-series web servers access log data or network traffic. Since outlier detection is commonly considered ...
0
votes
0answers
7 views
Large unsupervised problem (clustering). Where do I start?
I’m working on a dataset that is about 5,000,000 observations with about 500 variable. My goal is to find “some interesting things” about the data. I,e correlation between variables, similarities, ...
0
votes
0answers
13 views
AutoEncoder and Unsupervised Clustering
I am working on a dataset of ~300 samples with ~5000 data-points each - ranged between 0 and 100. I am interested in:
Group samples for similarity;
Find the differences between groups;
Would make ...
1
vote
1answer
55 views
Is it possible to find cluster centroids in kernel K means?
Suppose ${x_1, \ldots, x_N}$ are the data points and we have to find $K$ clusters using Kernel K Means.
Let the kernel be $Ker$ (not to confuse with $K$ number of clusters)
Let $\phi$ be the implicit ...
0
votes
1answer
49 views
Hierarchical clustering and Dendrogram interpretation
I'm quite new to cluster analysis and I was trying to perform a hierarchical clustering algorithm (in R) on my data to spot some groups in my dataset. Initially, I tried with the k-means, with the ...
0
votes
0answers
11 views
Blind source separation using FastICA
Reading the example by scikit-learn on how to use the FastICA function, I couldn't understand the following plot:
In the third plot ("ICA recovered signal") - why is the magnitude different ...
0
votes
0answers
13 views
How to cluster similar images with tensorflow?
I have a set of photographs and I would like to cluster based on their similarity in order to identify the ones that look alike and eventually select the best one among each cluster. I would like to ...
1
vote
0answers
16 views
How does the coloring of dendrograms in SciPy work?
So I am clustering my data using linkage extensions. When I plot the diagrams of the dendrograms scipy chooses to color branches in different colors according to a "color threshold". As I ...
0
votes
0answers
9 views
Technique for clustering bivariate data with no-overlap on one variable?
I am looking for a technique of clustering bivariate data. Basically, I have X and Y and I want to divide the data into k groups. However, on one variable, let's say Y, there can be no overlap of ...
1
vote
0answers
37 views
How to learn a neural network based on a cost function in an unsupervised way?
I need to learn a neural network that predicts L (see equation below). So based on three points from a discrete trajectory I want to learn L by minimizing the equation below, which is a physical ...
2
votes
1answer
77 views
KMeans clustering - can inertia increase with number of clusters
I am doing kmeans clusters on sales data and i see that inertia increases for the initial increase in the number of clusters. Can you please explain why that happens?
I am doing Batched Kmeans for the ...
0
votes
0answers
26 views
Techniques to measure similarities between instances and a dataset
I'd like to get a similarity measure between my dataset and a set of instances. For example, let's say I have a dataset with 4 features and n rows -
...
1
vote
1answer
92 views
unsupervised anomaly detection on sparse data
Given that I have a very sparse data matrix with continuous features, like this dataframe for example
...
1
vote
1answer
41 views
Confused between K-Means and Hierarchical Clustering for 9 different categories
I am trying to classify 9 different species of elephants into clusters using unsupervised learning. I have the following data about them:
Their height
Eye Colour
Sound they produce in decibel (dB)
I ...
0
votes
0answers
7 views
Is there a machine learning method for matching similar groupings?
I have a problem where I have rows/samples that are grouped together and each sample has a specific label (my data is genetic with genes being the samples and they are grouped together in the genome ...
0
votes
0answers
11 views
Is it viable to use ML models sequentially in a pipeline?
I have a machine learning model that predicts whether a gene is likely to cause a disease (the prediction is a probability score for a gene between 0 to 1, so a 1 score gene causes the disease and a 0 ...
0
votes
0answers
22 views
hierarchical clustering with categorical data
I have a dataset of 10K patients (row variables) with 10 disease conditions (such as heart condition, asthma, diabetes etc) along the columns. All the disease conditions are binary variables (yes/no).
...
0
votes
0answers
19 views
Outlier detection using 2D spatial information
I have a list of sensor measurements for air quality with geo-coordinates, and I would like to implement outlier detection. The list of sensors is relatively small (~50).
The air quality can gradually ...
0
votes
0answers
24 views
Extract Keyword/Concept From Column Description Using NLP
Suppose in my database, each table has a description associated with each column and I want to further extract keyword or key concept from the description. For example,
mean of transaction amount in ...
0
votes
0answers
11 views
How to determine a binary classifier threshold for unsupervised algorithms?
I am trying to create a model that can distinguish between normal and anomalous data. The training dataset contains non-anomalous data only. I am using an autoencoder that gets trained on this data to ...
2
votes
0answers
25 views
Is Normalized Mutual Info Score equivalent to V-measure when normalized by arithmetic mean
According to sklearn.metrics.v_measure_score, it says
This score is identical to normalized_mutual_info_score with the 'arithmetic' option for averaging.
In the ...
1
vote
0answers
13 views
General implications of low entropy for a dataset
I am fairly familiar with entropy, which quantifies uncertainty/surprisal of a random variable. In my case, I have a corpus where I can use empirical word frequencies to estimate entropy of the entire ...
1
vote
1answer
39 views
K means clustering breakup---galaxy spectrum data set
I have a spectrum data set (total 22000). Similar to an electronic wave data, two dimensional (Flux vs Wavelength). A typical set of wavelength plot looks like below
Now I am doing kmeans on this ...
0
votes
0answers
24 views
K-Means Clustering Problem in large 3D image CAD datasets
I have 1 million unclassified cad images in one folder and I need to cluster those images according to its similarity in 350 cluster folders, i.e k=350.
Out of 350 folders after clustering,
There are ...
0
votes
0answers
27 views
Clustering algorithm for a coordinate-based matrix
I have $1000$ scenarios, each of which is composed of $5$ users' coordinates $(x_i,y_i), \forall i \in \{1,\dots,5\}$. Now, based on users' coordinates, I want to cluster these $1000$ scenarios into ...
0
votes
0answers
10 views
Explanation of Excess Mass(EM)
I was researching on evaluation metrics to understand the performance of unsupervised anomaly detection algorithms and I came across this paper
The author suggests that EM and MV based numerical ...
0
votes
0answers
19 views
Clustering DBSCAN's parameter Epsilon: How is eps related to scale of data being clustered?
How is scale of eps related to data to be clustered in DBSCAN?
e.g. in image of 1024x1024, we have points as:
...
0
votes
0answers
11 views
Best way to suggest 'n' number of users based on demographic, interests data
I have to build a recommendation engine which suggests 'n' number of similar users to a user.
I tried to implement this in user-user recommendation system methodology and unsupervised learning.
Theres ...
1
vote
3answers
60 views
Unsupervised Anomaly Detection with groups
Let's say we are a bank and are interested in catching fraudulent customers. We gather ~100.000 independent samples of 40 independent variables and 4 are behavioral variables (what a customer does).
...
1
vote
0answers
13 views
autoencoders for radiographs - do watermarks affect the performance considerably
I have to implement an autoencoder to reconstruct the input radiographs and do unsupervised feature learning in the process. However, the radiographs that I have contain some watermarks like X-ray ...
2
votes
0answers
19 views
How can I cluster sequential data?
Suppose that I have a sequence of vectors $y_n \in \mathbb{R}^m$ for $n \in \{1, \dots, N\}$. My goal is to divide $y_n$ in $K$ clusters and want my clusters to satisfy the following conditions:
Each ...
1
vote
1answer
41 views
is k-means generalizable at any distance? [duplicate]
The classical version of k-means uses the Euclidean distance in the first step, and the arithmetic mean (the value center) in the second step. Is k-means generalizable to other distances and other ...
0
votes
1answer
23 views
Why doesn't t-SNE need the labels before visualization?
In the original Maaten and Hinton paper, they explicitly say that the class membership is not used by the t-SNE calculations, only for picking colors in the plot.
For all of the data sets, there is ...
0
votes
0answers
18 views
What is the statistical relevance of gamma in k-prototypes algorithm and why is it related to the standard deviation of the numeric columns?
The k-prototype algorithm uses gamma to provide weight to the categorical features. I have a few queries regarding it :
Why is there no upper limit to it? Should it not be (1-gamma) such that gamma ...
0
votes
0answers
20 views
Fixing the parameters of the variational distribution in Expectation-Maximization
Consider directed graphical model $z \to x$ (with $z$ unobserved and $x$ observed). The evidence lower bound on the log-likelihood $\log p(x) = \log \sum_z p(x, z; \theta)$ for parameters $\theta$ (...
1
vote
0answers
15 views
Best practice/Ideas for clustering Event Sequence Embeddings?
My dataset consist of around 40 000 samples of event sequences.
Sample of data
[[Event 1, Event 2, Event 4, Event 5],
[Event 1, Event 3, Event 4],
[...]]
I ...
1
vote
0answers
20 views
What is the meaning of noise in a dataset with no dependant variable?
My understanding of noise & signal comes from the context of bias-variance tradeoff in supervised methods.
But given a dataset with no dependant variable, how do you define noise? & how do you ...
2
votes
1answer
30 views
Updating an unsupervised model but retaining similarity
In my example I am using topic modelling (specifically a version of LDA) although I think avenues for exploring this could relate to other unsupervised techniques like clustering.
I train a model and ...
1
vote
2answers
38 views
How can PCA maximise variance after I standardise all predictor variance = 1?
I have been reading about Principal Components Analysis, and I think it is in general trying to extract as much "variance" out of the predictors $ \vec{X} = (X_1, X_2, ..., X_n)$ by ...
0
votes
0answers
17 views
Latest research and explanation on how semi-supervised learning is performing better than supervised?
So in AAAI 2020 also semi-supervised learning is given the push. There are some intuitive reasoning provided by people but since the research is so fast, I wanted to know actually what is the latest ...
