Questions tagged [unsupervised-learning]

Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.

460 questions
Filter by
Sorted by
Tagged with
220 views

Heuristics for unsupervised or semi-supervised approaches to GIS coordinate data

I have a more conceptual/heuristic question about how to go about formulating a problem in order to take a semi- or unsupervised method of solving it. I'm working on a project with data collected ...
12 views

Can NMF assign probabilities to the topics it outputs?

It's my understanding that only LDA can assign probabilities to words within each topic that it discovers since it's a probabilistic graphical model politicians 0.05 united states 0.10 obama 0.20 ...
24 views

Similarity measure/metric for long time series

I'm looking for a similarity measure/metric to cluster long time-series datasets. I feel that Euclidean distance won't do any good for my application, for it is not robust enough to detect patterns ...
162 views

How do i display the first principal component of an image after doing PCA (using SVD)?

Suppose i have an 10 images with 100x100 pixels. I have already converted the data into a 10x10000 dataset, subtracted the mean and performed SVD to get the eigenvectors and eigenvalues. Now i want to ...
8 views

n for Hopkins statistics using get_clust_tendency

I started to use the Hopkins statistics to establish, if a dataset is 'clusterable'. I am using the following code - taken from here: ...
32 views

Running k-means clustering with k = 2 recursively on clusters greater than a certain size

Does it make sense to run k-means with k (number of clusters) of 2, and then for every cluster bigger than N, run k-means again with k = 2? We can then keep doing it until we have all clusters of size ...
15 views

Anomaly Detection Without a Baseline

I am attempting to find anomalies in accounting data (similar to this study: https://arxiv.org/pdf/1709.05254.pdf). I don't have any labeled data, so this attempt needs to be unsupervised. I am having ...
13 views

Can I use KernelPCA after using TruncatedSVD before clustering?

I am working on a project at a company where I have to make clustering/unsupervised model. The data I am working on is very sparse with high dimensions and after some research, I found out ...
22 views

Extract data from mainly unstructured sets and derive risk metrics out of those

I have the following question (this was a real life problem): Q: Extract data from mainly unstructured sets and derive risk metrics out of those. From what you know or imagine about the data ...
406 views

Training and testing an autoencoder on very sparsely populated data

I am exploring the possibility of using a deep autoencoder neural net to build a recommender system. I am firstly testing the model's performance on the traditionally used benchmark of the Movielens ...
297 views

When does my autoencoder start to overfit?

I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has ...
169 views

Difference between Hartigan & Wong Algo to Lloyd's algorithm in K-means clustering

In the iterations of Hartigan and Wong Algo of K-Means clustering, If the centroid is updated in the last step, for each data point included, the within- cluster sum of squares for each data point if ...
323 views

Validating Clustering by Considering the Ratio of Intra-cluster to Inter-cluster Distance

I'm trying to evaluate a clustering method by looking at the ratio of the mean intra-clustering distance (the average distance between points in the same cluster) to the mean inter-cluster distance (...
235 views

Methods for unsupervised subset selection on categorical data

I am new to this. I have a set of survey data with 18 questions (columns/features) with 165 observations. Responses are ternary: True, False, Don't Know. Each question has a correct response, which ...
48 views

gensim LdaModel - How to reduce the number of words in each topic?

I'm trying to get more sparse topics (Less overlaps between different topics). https://radimrehurek.com/gensim/models/ldamodel.html I know it should be determined by the alpha parameter. I've ...
419 views

Prediction after PCA and K-Means

I have a data set with a large amount of features. I'm applying PCA on it in order to run it through K-means, to discover clusters in my data set. I'd like to know what is the best practice to make ...
13 views

Usage of VIF in unsupervised model

I'm working on building an unsupervised model for real time anomaly detection based on the concept of Randomized Matrix Sketching (http://www.vldb.org/pvldb/vol9/p192-huang.pdf) which involves ...
14 views

24k views

Unsupervised, supervised and semi-supervised learning

In the context of machine learning, what is the difference between unsupervised learning supervised learning and semi-supervised learning? And what are some of the main algorithmic approaches to ...
1k views

Feature selection clustering customer segmentation

based on customer data I want to perform a clustering using different clustering algorithms (K-Means, Expectation Maximization, etc.) in R. The most attributes were engineered pursuing the goal to be ...
125 views

Deep learning models for unsupervised semantic segmentation

I am working on semantic segmentation for satellite images using keras and python. It is my understanding that popular models like U-Net require mask images (labels). Are there any unsupervised deep ...
8 views

Is there a representer theorem for unsupervised learning (to justify kernel density estimation)?

In supervised learning, we get a representer theorem by considering regularized losses of the following form: In Kernel Density Estimation, we simply directly assume densities of the form Could this ...
1k views

Unsupervised Outliers detection on time series

So I am looking ways to improve my current implementation of detecting outliers in work schedule. My data set is badge swipes for people. The current implementation finds outliers on in-times and out-...
15 views

Predicting user behaviour based on transactional data - flagging “risky” behaviour

Firstly, I'm not sure if this is the right instance of StackOverflow to post on so feel free to ask me to put it elsewhere. I am exploring the concepts of clustering and "unsupervised" learning for ...
33 views

Unsupervised Clustering

My research is about comparing K-means and DBSCAN, and Im using unsupervised learning method in clustering. Is it true that the number of cluster in K-means is also the same number as the unique ...
25 views

Inference can be the goal of an unsupervised learning method or a semi-supervised learning method or even more of a reinforcement learning method?

I am new to machine learning, and I am reading a pair of machine learning books. These references talk about 2 different learning approaches: Prediction and inference, I understand the difference ...
36 views

Meaning of Probability Distributions in RBMs

I'm new to machine learning, and am trying to understand some of the basics of Restricted Boltzmann Machines. Unfortunately, I don't have a background in statistics yet beyond a basic understanding, ...
29 views

R - high dimension data using k means clustering [closed]

The dataset is 1000(observations) x 700(variables), After using pca to do dimension reduction, PC150 explained 85% Variance, so I use this (1000 x 150) data to do k means clustering. This code was ...
30 views

Hierarchical clustering for aggregrated features at higher thresholds/levels?

I am trying to use clustering on certain data. The data itself has three natural levels: at the lowest level the elements are fundamental building blocks, at the second level these fundamental ...
63 views

Scaling data with different importance

I have 9 attributes: x1,x2,x3,x4,...,x9 and I know that the attributes x9 must have the same value in a cluster and the attribute X1 have more importance than others (x2,...,x8) I'm using Euclidean ...
499 views

Optimizing cumulative lift in classification

Suppose I have a business problem where I can reach out to 10% of my customers to prevent them from churning. I want to capture as much of the high risk customers I can. Let's say I'm tuning a random ...