Questions tagged [unsupervised-learning]

Finding hidden (statistical) structure in unlabelled data, including clustering and feature extraction for dimensionality reduction.

Filter by
Sorted by
Tagged with
2
votes
1answer
220 views

Heuristics for unsupervised or semi-supervised approaches to GIS coordinate data

I have a more conceptual/heuristic question about how to go about formulating a problem in order to take a semi- or unsupervised method of solving it. I'm working on a project with data collected ...
0
votes
1answer
12 views

Can NMF assign probabilities to the topics it outputs?

It's my understanding that only LDA can assign probabilities to words within each topic that it discovers since it's a probabilistic graphical model politicians 0.05 united states 0.10 obama 0.20 ...
-1
votes
2answers
24 views

Similarity measure/metric for long time series

I'm looking for a similarity measure/metric to cluster long time-series datasets. I feel that Euclidean distance won't do any good for my application, for it is not robust enough to detect patterns ...
0
votes
1answer
162 views

How do i display the first principal component of an image after doing PCA (using SVD)?

Suppose i have an 10 images with 100x100 pixels. I have already converted the data into a 10x10000 dataset, subtracted the mean and performed SVD to get the eigenvectors and eigenvalues. Now i want to ...
-1
votes
1answer
8 views

n for Hopkins statistics using get_clust_tendency

I started to use the Hopkins statistics to establish, if a dataset is 'clusterable'. I am using the following code - taken from here: ...
1
vote
2answers
32 views

Running k-means clustering with k = 2 recursively on clusters greater than a certain size

Does it make sense to run k-means with k (number of clusters) of 2, and then for every cluster bigger than N, run k-means again with k = 2? We can then keep doing it until we have all clusters of size ...
-1
votes
2answers
15 views

Anomaly Detection Without a Baseline

I am attempting to find anomalies in accounting data (similar to this study: https://arxiv.org/pdf/1709.05254.pdf). I don't have any labeled data, so this attempt needs to be unsupervised. I am having ...
0
votes
1answer
13 views

Can I use KernelPCA after using TruncatedSVD before clustering?

I am working on a project at a company where I have to make clustering/unsupervised model. The data I am working on is very sparse with high dimensions and after some research, I found out ...
0
votes
0answers
22 views

Extract data from mainly unstructured sets and derive risk metrics out of those

I have the following question (this was a real life problem): Q: Extract data from mainly unstructured sets and derive risk metrics out of those. From what you know or imagine about the data ...
0
votes
1answer
406 views

Training and testing an autoencoder on very sparsely populated data

I am exploring the possibility of using a deep autoencoder neural net to build a recommender system. I am firstly testing the model's performance on the traditionally used benchmark of the Movielens ...
1
vote
1answer
297 views

When does my autoencoder start to overfit?

I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has ...
0
votes
1answer
169 views

Difference between Hartigan & Wong Algo to Lloyd's algorithm in K-means clustering

In the iterations of Hartigan and Wong Algo of K-Means clustering, If the centroid is updated in the last step, for each data point included, the within- cluster sum of squares for each data point if ...
1
vote
1answer
323 views

Validating Clustering by Considering the Ratio of Intra-cluster to Inter-cluster Distance

I'm trying to evaluate a clustering method by looking at the ratio of the mean intra-clustering distance (the average distance between points in the same cluster) to the mean inter-cluster distance (...
1
vote
1answer
235 views

Methods for unsupervised subset selection on categorical data

I am new to this. I have a set of survey data with 18 questions (columns/features) with 165 observations. Responses are ternary: True, False, Don't Know. Each question has a correct response, which ...
0
votes
1answer
48 views

gensim LdaModel - How to reduce the number of words in each topic?

I'm trying to get more sparse topics (Less overlaps between different topics). https://radimrehurek.com/gensim/models/ldamodel.html I know it should be determined by the alpha parameter. I've ...
0
votes
3answers
419 views

Prediction after PCA and K-Means

I have a data set with a large amount of features. I'm applying PCA on it in order to run it through K-means, to discover clusters in my data set. I'd like to know what is the best practice to make ...
0
votes
0answers
13 views

Usage of VIF in unsupervised model

I'm working on building an unsupervised model for real time anomaly detection based on the concept of Randomized Matrix Sketching (http://www.vldb.org/pvldb/vol9/p192-huang.pdf) which involves ...
0
votes
1answer
14 views

Can I use the Silhouette to measure quality of clusters in different dimensions?

Can I use the Silhouette to measure quality of clusters in different dimensions? For example, let's say we run kmeans for some $k$ using 6 features of the dataset. Mark the resulted silhouette as $...
2
votes
1answer
280 views

Unsupervised Learning on Multilevel/Multidimensional Data

I am working on a case-control study, where I for each individual have high dimensional data (like illustrated in the image). I would like to do both PCA analysis and Clustering on this data, but it ...
0
votes
1answer
298 views

Cluster data based on similarity, but split by one feature (Python, R)

Is there a way to do unsupervised clustering based on similarity (like all other methods), but create clusters by splitting on just one (or specific) features? For example, I have customer data with ...
0
votes
1answer
334 views

Machine Learning - Aggregate granular cluster predictions on denormalized data

I have a question that occurred when thinking about the following use case: A bank wants to group their customers into segments using the database tables 'customer', 'account' and 'transaction'. The ...
2
votes
1answer
238 views

Image feature extraction using an Autoencoder combined with PCA

Background: I have fairly large dataset of biomedical images (around 10,000 images) of 1920x1920 pixels (after cropping parts of black borders out). My task is to extract the 200 most important ...
0
votes
0answers
16 views

LDA: weight distributions of inferred documents

I have trained a two-topic Latent Dirichlet Allocation (LDA) model on a corpus and I am now inferring on a test corpus (the nature of the corpus is irrelevant). During inference, for each new document ...
0
votes
3answers
254 views

Why are decision trees considered supervised learning?

It seems to work similar to clustering algorithms, where data does not have to be labeled, and the algorithm creates it's own labels/groups based on feature similarities...
9
votes
1answer
167 views

t-SNE with mixed continuous and binary variables

I am currently investigating the visualisation of high-dimensional data using t-SNE. I have some data with mixed binary and continuous variables and the data appears to cluster the binary data much ...
6
votes
1answer
333 views

What are the votes in R's unsupervised random Forest?

I’m trying to better understand unsupervised random forests. An important part of understanding unsupervised random forests is being able to assess how good / appropriate a given forest is. For ...
0
votes
0answers
29 views

Unsupervised classification of images

Assuming I have a dataset of images from two similar classes, for example let's say 95% Labradors and 5% of Chihuahuas and I want to make a classifier. The point is that I need to find the anomalies (...
1
vote
1answer
578 views

Unsupervised learning methods on unlabeled data?

I'm facing with a challenge of unsupervised classification of unlabeled data. The case is, I have circa 1.2 million vehicle warranty claims, and must develop a classification model to tell whether ...
0
votes
0answers
25 views

Looking for ways to transform time-series data recorded from object movement into equation describing the movement direction of the object

Looking for some time-series data transformation advice! I want to know what's the best way to transform data of 9-tuples time series data of IMU (Inertia Measurement Unit) sensor, recorded from a ...
1
vote
0answers
43 views

Recognizing whether a written and spoken number is the same

For our ML assignment we have three datasets. The challenge is about checking whether a written and spoken number refer to the same number. We're using the MNIST dataset with handwritten numbers, and ...
30
votes
2answers
40k views

Choosing the right linkage method for hierarchical clustering

I am performing hierarchical clustering on data I've gathered and processed from the reddit data dump on Google BigQuery. My process is the following: Get the latest 1000 posts in /r/politics ...
1
vote
0answers
29 views

How to measure correlation between two groups of variables?

I have a data set that contain 75 variables of football players . These 75 variables basically measures two different types of information. 30 of those variables related to bio metric information ...
5
votes
1answer
7k views

K-means Mahalanobis vs Euclidean distance

I currently am trying to cluster "types" of changes on bitemporal multispectral satellite images. I applied a thing called a mad transform to both images, 5000 x 5000 pixels x 5 bands. Each band is a ...
1
vote
1answer
430 views

clustering for categorical data with one column for observations

I'm trying to cluster a dataset using 4 variables, all of which are categorical variables. I'd also like to include another numerical variable that's actually the number of observations of another ...
0
votes
0answers
11 views

Metric learning with respect to an outcome

Suppose I have $n$ datapoints in $p$-dimensional space, and the $p$ variables are highly heterogenous. That is, there is no natural way to combine them. Some are ordinal, some one-hot, some continuous,...
26
votes
3answers
24k views

Unsupervised, supervised and semi-supervised learning

In the context of machine learning, what is the difference between unsupervised learning supervised learning and semi-supervised learning? And what are some of the main algorithmic approaches to ...
0
votes
2answers
1k views

Feature selection clustering customer segmentation

based on customer data I want to perform a clustering using different clustering algorithms (K-Means, Expectation Maximization, etc.) in R. The most attributes were engineered pursuing the goal to be ...
0
votes
1answer
125 views

Deep learning models for unsupervised semantic segmentation

I am working on semantic segmentation for satellite images using keras and python. It is my understanding that popular models like U-Net require mask images (labels). Are there any unsupervised deep ...
1
vote
0answers
8 views

Is there a representer theorem for unsupervised learning (to justify kernel density estimation)?

In supervised learning, we get a representer theorem by considering regularized losses of the following form: In Kernel Density Estimation, we simply directly assume densities of the form Could this ...
0
votes
1answer
1k views

Unsupervised Outliers detection on time series

So I am looking ways to improve my current implementation of detecting outliers in work schedule. My data set is badge swipes for people. The current implementation finds outliers on in-times and out-...
1
vote
0answers
15 views

Predicting user behaviour based on transactional data - flagging “risky” behaviour

Firstly, I'm not sure if this is the right instance of StackOverflow to post on so feel free to ask me to put it elsewhere. I am exploring the concepts of clustering and "unsupervised" learning for ...
-1
votes
2answers
33 views

Unsupervised Clustering

My research is about comparing K-means and DBSCAN, and Im using unsupervised learning method in clustering. Is it true that the number of cluster in K-means is also the same number as the unique ...
0
votes
1answer
25 views

Inference can be the goal of an unsupervised learning method or a semi-supervised learning method or even more of a reinforcement learning method?

I am new to machine learning, and I am reading a pair of machine learning books. These references talk about 2 different learning approaches: Prediction and inference, I understand the difference ...
0
votes
2answers
36 views

Meaning of Probability Distributions in RBMs

I'm new to machine learning, and am trying to understand some of the basics of Restricted Boltzmann Machines. Unfortunately, I don't have a background in statistics yet beyond a basic understanding, ...
0
votes
2answers
29 views

R - high dimension data using k means clustering [closed]

The dataset is 1000(observations) x 700(variables), After using pca to do dimension reduction, PC150 explained 85% Variance, so I use this (1000 x 150) data to do k means clustering. This code was ...
1
vote
1answer
30 views

Hierarchical clustering for aggregrated features at higher thresholds/levels?

I am trying to use clustering on certain data. The data itself has three natural levels: at the lowest level the elements are fundamental building blocks, at the second level these fundamental ...
2
votes
1answer
63 views

Scaling data with different importance

I have 9 attributes: x1,x2,x3,x4,...,x9 and I know that the attributes x9 must have the same value in a cluster and the attribute X1 have more importance than others (x2,...,x8) I'm using Euclidean ...
1
vote
1answer
499 views

Optimizing cumulative lift in classification

Suppose I have a business problem where I can reach out to 10% of my customers to prevent them from churning. I want to capture as much of the high risk customers I can. Let's say I'm tuning a random ...
0
votes
1answer
83 views

Issue in evaluating the performance of my “clustering algorithm” using NMI, ARI when the “ground truth” is available? [duplicate]

(**Edited the question after the initial comments) Suppose, Ground_truth_data = [1, 1, 1, 1, 1, 1, 1]; Clustering_result = [1, 1, 1, 1, 1, 1, 2]; Here, as you can see, there are "7" instances of ...
1
vote
0answers
23 views

How to interpret LDA (Latent Dirichlet Allocation)?

Say I want to run topic modeling with LDA on The 20 newsgroups text dataset. So basically a dataset with texts where every text belongs to one of 20 categories. I want the LDA to split the documents ...