0
votes
0answers
7 views

Understanding differences between large and small dimensional data when implementing algorithms

I'm working on a local outlier factor implementation based on the wikipedia entry : http://en.wikipedia.org/wiki/Local_outlier_factor This article seems to explain it in just two dimensional data. ...
0
votes
0answers
37 views

How to represent outliers for multi dimensional data (local outlier factor)

Below graph taken from http://en.wikipedia.org/wiki/Local_outlier_factor displays "LOF scores : LOF image : This is great for two dimensional data but what about data > than two dimensions. How ...
1
vote
1answer
47 views

How to evaluate a clustering/unsupervised learning problem with massive amounts of data, with labels only for a small fraction of points

I'm wondering if anybody can point me to work on the evaluation of unsupervised learning where there are a very large (say hundreds of millions) number of points and manual labelling can only ever be ...
0
votes
1answer
25 views

Cluster migration visualization

I have asked a very similar question at the Latex forum here, but in order to address the part of my question where I ask if there is a better way of visualizing the data I have, I wanted to cross ...
1
vote
2answers
28 views

How to compare two clusterings generated by two clustering approaches

I am currently working on a modification of a clustering algorithm to suit my problem domain. I want to know which methods are available for me to compare the centroids generated from the two ...
0
votes
0answers
20 views

Interpreting R results, are the data multivariate normal?

I ran "mvn" using the "mclust" package in R using the following codes: mvn("EEE", data[,18:22], prior = NULL, warn= NULL) I am having trouble figuring out how to ...
0
votes
1answer
38 views

R codes for variation of information criterion using “mclust”

I am developing model-based clustering. First, I developed model-based clustering in R using "mclust." Next, I wanted to take 75% of the sample, re-run model-based clustering and compare the ...
0
votes
1answer
25 views

Is my understanding of how to calculate the reachability distance in local outlier factor correct?

Reading lof implementation at : http://www.cse.ust.hk/~leichen/courses/msc-it5210/lectures/LOF_Example.pdf the local reachability distance is given as : I don't fully understand this equation as ...
0
votes
1answer
58 views

How to determine which variable or combination of the variables are affecting to the predictor variable?

I have one dependent variable name as "win ration" of the deal contested and more than 30 independent variables, all are categorical variable name as role of the customer, geo, region, and 27 ...
6
votes
2answers
159 views

Clustering a noisy data or with outliers

I have a noisy data of two variables like this. ...
1
vote
1answer
116 views

What are basic differences between Kernel Approaches to Unsupervised and Supervised Machine Learning

I got nice graphical representation of Machine learning for clustering / classification. Source: Kernel Approaches to Unsupervised and Supervised Machine Learning by Sun-Yuan Kung Here are my ...
0
votes
0answers
45 views

Profile variable in collaborative filtering

I'm trying to create a recommendation system based on purchases. I did some tests and I found that for some groups of customers, the recommender works very well, but not for others. How can I ...
2
votes
1answer
39 views

One-class Classification of multidimensional vectors

I have a m x k User-feature matrix (m >> k) obtained by factorizing an original User-websites matrix (m x n) that has #page views as entries. Additionally, there are users (say r) who have been ...
-1
votes
1answer
29 views

Metrics for cluster evaluation

I make a set of clusters using some clustering algorithm. Precision, Recall, F Measure, Fallout and RI of individual clusters are calculated for testing the performance. How do I calculate the average ...
0
votes
0answers
15 views

Turning MiniBatchKMeans into Fuzzy MiniBatchKMeans

I'm using Scikit-Learn, which has an implementation of MiniBatchKMeans. I'm very inexperienced with ML, so I'm wondering how (if ...
1
vote
1answer
60 views

Best metric for evaluation of mixture-of-Gaussian clusters on big-data

I have made a new algorithm that is specifically crafted for clustering very large datasets. In order to document it as a research paper, I have to choose one or two internal (no-label) cluster ...
0
votes
1answer
53 views

Identifying subsets for outlier detection in local outlier factor

I am trying to gain better understanding of the idea of local outliers (as discussed in this pdf) and how the function is implemented. Here are the key passages from the pdf: Local outliers: ...
2
votes
1answer
50 views

Selecting number of clustering classes automatically

I am working in text clustering. I would like to find a way to identify the number of classes for the clustering process automatically rather than proving the number of class manually. Is their any ...
1
vote
0answers
16 views

Heuristics for unsupervised or semi-supervised approaches to GIS coordinate data

I have a more conceptual/heuristic question about how to go about formulating a problem in order to take a semi- or unsupervised method of solving it. I'm working on a project with data collected ...
0
votes
0answers
32 views

BOW prediction of cluster for training data

I am creating a bag of visual words for classification of videos. I am not using SURF descriptors and that is why I couldn't use OpenCV's BOWImgDescriptorExtractor ...
2
votes
2answers
60 views

Cluster Data based on Distribution

I have a list of diseases for my research. For each disease, I have a list of ages for the diseases. "Breast Carcinoma" may be a list of [1,2,2,3,4,5,5,5,5,5] while "Adrenal Cortex Neoplasms" maybe be ...
2
votes
1answer
123 views

Is there a decision-tree-like algorithm for unsupervised clustering?

I have a dataset consists of 5 features : A, B, C, D, E. They are all numeric values. Instead of doing a density-based clustering, what I want to do is to cluster the data in a decision-tree-like ...
0
votes
0answers
24 views

clustering versus projection - what are the best example/scenario to explain their differences

I am dealing with non statistician who know are crunching data but don't have a deep understanding of statistics. I am trying to introduce them to non-matrix factorization methods but it has been ...
1
vote
0answers
22 views

ML Bank transactions assignement to invoices

In a effort to reduce human intervention, I'm trying to optimize the process of assigning bank transactions to invoices. This task should be done once every year, so we can assume our dataset won't ...
-3
votes
2answers
69 views

Question about KNN algorithm

Does KNN algorithm have a centroid as k-means? Is there a way to obtain the centroid for the classified data by KNN? Is there a way to compare SVM classification with KNN classification?
8
votes
1answer
106 views

Nonparametric mixture model and clusters

I have a question about clusters that I am contemplating to treat with a nonparametric mixture approach (I think). I am working on the explanation of human comportment. Each row of my database ...
0
votes
0answers
36 views

Assessing clustering statistical significance

I read a good tutorial on the p-value. Everything there is clear for me, except how I can define the null hypothesis. I found an example (on page 6) in this paper using p-value to judge the ...
60
votes
6answers
6k views

Why is Euclidean distance not a good metric in high dimensions?

I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ...
1
vote
0answers
42 views

Is there a way to perform SVD in a sequential manner?

My neurology experiment has a spike detector outputting 40 sample long spike waveforms. I'm using a dictionary method for sorting the spikes in real time. To ...
1
vote
0answers
50 views

Clustering using longitude, latitude, and some other variable

I am hoping someone can point me in the right direction with this problem I am having. I am trying to cluster geographical areas (basically using latitude and longitude as the zip code centroid) ...
0
votes
1answer
68 views

Clustering structured data: Assessing the similarity of documents that appear in tree structure

Usually when performing text document clustering, similarities across documents are assessed based on the lexical content of documents. But, in my problem, I wish to consider both the lexical content ...
-1
votes
1answer
96 views

Clustering large movie dataset using k-medoids?

I have to cluster a movie dataset of 10000 movies. A movie has attributes like Genres, Actors, Directors, Year. Earlier I thought that we can use a simple clustering algorithm like k-medoids and the ...
0
votes
0answers
73 views

Feature Selection for look alike modeling using k-NN

I have a list of items and various parameters for each items. For each item on my list i need to identify items which are similar to the item from my whole population . I am planning on using K-NN ...
2
votes
1answer
69 views

How can I evaluate the performance of a system that generates word clusters?

The word2vec tool uses deep learning to compute vector representations of words. They've mentioned that - "The word vectors can be also used for deriving word classes from huge data sets. This is ...
0
votes
1answer
65 views

Performing hierarchical clustering on a large data set

I have been applying complete linkage on about 5,000 points using matlab with no problem. I want to extend this method to much more elements. It would take me a long time to process my data to test ...
0
votes
1answer
25 views

Evaluation indexes hypothesis for clustering

I read on the cluster analysis page of wikipedia: For example, k-means clustering can only find convex clusters, and many evaluation indexes assume convex clusters. On a data set with non-convex ...
0
votes
0answers
19 views

k-way MinMax spectral clustering

Is there a k-way MinMax Cut Spectral Clustering which can be easily implemented? In the spectral clustering tutorial I found only 2-way MinMax cut.
1
vote
0answers
29 views

Comparison of cophenetic correlation coefficients on different data sets

I applied the same hierarchical clustering (weighted) on two data sets: The first is a 'raw' data set, on which I didn't do anything, and the second on the same data set after I filtered it by ...
0
votes
0answers
22 views

processing a sound file for analysis of different spoken languages

So i have sound files for 5 languages by 2 person, thus my input data has 10 sound files. Now i want to cluster them based on the languages (thus 5 clusters) and not based on the speaker/voice ( ...
0
votes
0answers
82 views

Converting a spearman correlation to a euclidian dissimilarity

I am applying ward hierarchical clustering on a data set for which I have pairwise similarities. Since hierarchical clustering need a dissimilarity matrix, I am trying to convert my similarity matrix ...
1
vote
1answer
136 views

Understanding and Implementing a Dirichlet Process model

I am trying to implement and learn a Dirichlet Process to cluster my data (or as machine learning people speak, estimate the density). I read a lot of paper in the topic and sort of got the idea. ...
0
votes
0answers
24 views

What is data augmented by the additive inverse?

I am reading Biclustering of expression data (Cheng and Church, 2000) The paper is about the Cheng and Church biclustering algorithm and its main metric, the mean squared residue (MSR). It is said ...
0
votes
0answers
29 views

Converting a similarity to a dissimilarity [duplicate]

I am working on a clustering problem for which I have to manually choose the number of clusters. I have a visualization tool that helps me decide whether the clusters are good. In order to ...
1
vote
0answers
27 views

What is a shift bicluster?

I am reading A comparative analysis of biclustering algorithms for gene expression data (Eren, Kemal, et al. - 2013) When explaining the Cheng and Church method, it says that: MSR was shown to be ...
1
vote
0answers
47 views

Vector Quantization of heavy tailed distribution

I'm generating with Monte Carlo simulation some stock price $X$. Once I have the stock price sample, I want to cluster it with 100 points $\hat{X}$. My problem is that the error associate with my ...
0
votes
1answer
40 views

Data Conversion to Standard data format in hierarchical Dirichlet process

I'm trying to test the performance of posterior inference on a set of documents with hierarchical Dirichlet process for topic modeling. How can i convert my data (document) to standard data format ...
-1
votes
1answer
25 views

Converting FuzzykMeans to SphericalFuzzyKMeans?

I grabbed an implementation of FuzzyKMeans (FuzzyCMeans) from the nightly build of the Apache Commons Math library, but I now realize I need to use Cosine Similarity instead of the Euclidean Distance. ...
-2
votes
1answer
53 views

Clustering algorithms assigning probability values

I have a distance matrix for some data I want to cluster. However, I don't just want to assign elements to clusters, but I also want to assign a probability for each element to belong to each cluster. ...
2
votes
1answer
188 views

Run time analysis of the clustering algorithm (k-means)

I was reading some notes on ML and clustering and it claimed that the run time of clustering was O(kn) where k is the number of clusters and n is the number of points. I was wondering why this was ...
2
votes
1answer
83 views

Evaluating the clustering of a Kohonen UMatrix

Given a converged Kohonen feature map, how would one evaluate the clustering in terms of intra- and inter-cluster distances? Assuming that both the trained codebook vectors and Unified Distance ...