Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

0
votes
0answers
5 views

Quick Exploratory Analysis of Categorical Data

Does anyone know of a tool (preferably free) that does quick analysis of exploratory data mainly categorical with date. Using R and Python I can create time series and histograms, perform tests such ...
0
votes
0answers
9 views

Find input image (ID,passport) in imagesDB based on similarity

I would like to decide if image is exists on DB images (pictures of IDs,passport,Stu. card,etc) I thought of KNN alghorithem that will plot the K closest images. Options for distance metric: 1) sum ...
0
votes
1answer
15 views

identify nature of missingness for categorical variables

could you please give me some hints for identifying the nature of missingness for categorical variables' missing value? I mean, I gave a fast search on google scholar but I didn't find anything ...
0
votes
0answers
32 views

Unsupervised learning algorithems to detect anomaly in waves

I have a sample of graphs (more then 10000...). that look like in the image below: I am searching an Unsupervised learning algorithems thet can help me to detect Anomaly observations. Here what i ...
2
votes
2answers
25 views

Best algorithm for association rule mining

I am working on an application where I have to extract or identify association / correlation between different sets of items. An example would be say if a person buys shoes at a store, would he/she ...
0
votes
0answers
22 views

Finding genuine arrears and default arrears from rent payment patterns

I am currently working on some housing data - in particular analyzing the tenants' rent payment information and I am stuck on progressing with the following: I have to classify tenants based on their ...
1
vote
1answer
41 views
+50

How to categorize classifiers and matrix factorization methods?

I have a classification problem which is solved by a variety of methods. Among the methods are unsupervised methods, traditional classifiers and a supervised matrix factorization methods. The problem ...
0
votes
0answers
14 views

Create a density map with latitude and longitude

I am working on a data mining project for my class and would like to visualize some data. I basically have 20,000 instances and would like to create a density map based on house value. I have ...
-2
votes
0answers
18 views

Can we use kNN and k-mean at a same time?

I Get dataset of neighbours using kNN and then I want to apply k-mean on that dataset. By using this, is it possible that I get more accurate result? Is it logically correct that use kNN and then ...
0
votes
0answers
15 views

Determine performance in which subject improves overall performance

I have a dataset in .csv format as shown: ...
0
votes
0answers
17 views

input variables with different order of magnitude [duplicate]

I need to build a prediction model based on a data set with 5 different independent variables. The data set looks like as follows. The variables in col4 and ...
1
vote
1answer
57 views

Better in ROC AUC vs. better in PR AUC

I'm comparing two classification models by computing the area under ROC and Precision-Recall curves. However sometimes one model is better with AU-ROC but worse in AU-PR, and other times it's better ...
1
vote
0answers
35 views

How to normalize the data to [0, 1] in R with data similar to χ²-distribution without shrinking lower values too much?

I want to normalize the data to [0,1], but the distribution of this array is quite not regular, having large quantity of low values and small quantity of large values, almost 80% values of data are in ...
0
votes
0answers
27 views

SVM One-vs-One vs One-vs-ALL SVM

For an unbalanced dataset annotated by human annotators in which each item is assigned to different classes, what is the argument for and against using any of One-vs-One vs One-vs-ALL SVM ...
0
votes
0answers
11 views

Generating Labeled Training data from 2 data sources for Predictive Classifier

I am trying to build a predictive risk model classifier for an product (classifying good or bad). I am in the process of creating a training dataset. Here are the challenges I am facing. I have 2 ...
0
votes
0answers
15 views

Transforming frequency data into a rating system

I'm working on a project for fun using data (items from a persistent video game) I've gathered from the web. At the moment, the data consists of around 180,000 rows which will probably grow quite ...
0
votes
0answers
6 views

Acquring entity labeling/tagging data

I am interested in one data mining project. The input of the program is entities of certain types, e.g, Company. The output is the labels/tags that characterize the ...
0
votes
1answer
57 views

When normalization is counter-productive [duplicate]

Could you give me general examples of when normalization is not used properly and affects badly the classification accuracy, or when it is not needed?
0
votes
0answers
32 views

difference between machine learning and stastitical technique [duplicate]

Is there any difference between machine learning and stastitical techniques. I have searched a lot some researchers say that there are some overlap some are saying there is no difference.Can you give ...
2
votes
1answer
83 views

How can I remove multicollinearity from my logistic regression model?

I am working on Sales data. i have binary variable win/loss the opportunities and rest are the activities done by sales force (sales guys) with 40+ variables (different types of activities done for ...
1
vote
1answer
42 views

Why AUC-PR increases when the number of positives increase?

I asked a question earlier about comparing models using Precision-Recall AUC. One of the answers included the following statement: "The larger the fraction of positives in the data set, the larger the ...
1
vote
1answer
41 views

Ranking two models based on ROC-AUC and PR-AUC

I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed ...
2
votes
1answer
34 views

Is it correct to use Precision-Recall AUC in a balanced dataset situation?

I have a binary classification scenario with a dataset that is unbalanced (much more negatives than positives). When I train a classifier on this dataset I get a Precision-Recall AUC of 0.7. Then I ...
0
votes
0answers
28 views

How to calculate the area under the precision-recall curve for the random classifier?

I know that the random classifier score in ROC AUC (Area under the curve) is always 0.5. My question is: how to calculate the Area under the precision-recall curve for the random classifier?
-1
votes
0answers
36 views

Queries in GBM Model in R

I have the following questions pertaining to GBM Model in R. Question 1 : I have a mixture of factorial and numerical data. And Its a binary classification ...
0
votes
0answers
40 views

How to plot a precision-recall curve when doing cross-validation?

I'm using cross-validation to evaluate the performance of a classifier with scikit-learn and I want to plot the Precision-Recall curve. I found an example on ...
4
votes
1answer
92 views

Unintuitive interpretation of probabilities when doing logistic regression

The observations in my dataset can be split in two classes. The observations in class 1 are for sure correctly labeled. The observations that has been designated to class 2 have a huge percentage of ...
0
votes
0answers
30 views

clustering analysis of large amount of time series

I would like to cluster a set of time series, which are composed of around 50000 different time series. Are there established algorithms/package that can handle this scalability problem?
1
vote
0answers
55 views

Incremental aggregation of statistics

I need your suggestion to solve this problem: Given a Data Warehouse with e.g. 10 million entries, additional 1000 entries arrive each day. Rather than recomputing the desired aggregates, an ...
-1
votes
2answers
50 views

Will RandomForest model work well if the correlation lies between the given attributes

Currently I am working on the data mining project and I am using RandomForest classification model for that. I have a few queries in that. Will the RandomForest handle if there is correlation ...
0
votes
1answer
57 views

What is the convex hull in ROC curve?

I'm reading a paper about ROC and PR curves. They mentioned the ROC convex hull but they don't define it or say what it is. Can someone please tell me the meaning of it? What is a convex hull in ROC ...
2
votes
2answers
55 views

An intuitive meaning of the area under the PR curve?

Wikipedia says that an interpretation of the area under the ROC curve is: "the area under the curve is equal to the probability that a classifier will rank a randomly chosen positive instance higher ...
0
votes
1answer
31 views

build model with complicated types of feature variables

I have been asked to build a model to predict a life span of a material based on a couple of features. The features can be classed into the following categories: 1) The feature variables just have 0 ...
4
votes
2answers
146 views

What are some good examples of exploratory data analysis today?

Are there some papers published which illustrate EDA used to tackle substantial data problems? I am particularly looking for actual (current) data examples, where plots have been made and statistics ...
0
votes
0answers
27 views

How to measure distance between two elements in a web page

Suppose we want to measure how closely relevant is one word or phrase to another in a webpage, this can be easily seen by eyes whether they are near. But in an HTML document it could be tricky to ...
0
votes
0answers
13 views

M statistic and software ProtoArray Prospector

I would like to reproduce the statistical result in this paper: http://www.ncbi.nlm.nih.gov/pubmed/24947363 They said "the M-statistic test in ProtoArray Prospector was applied to the normalized ...
2
votes
3answers
116 views

Good sources to get papers in machine learning?

I'm a CS master student (will be a PhD in three months or so). Today I was at my superviser's office and he had a friend discussing some ideas for their research. Then they mentioned a paper that I ...
0
votes
0answers
8 views

Multiple combinations (ex drug-ADR) with the same unique case ID.

I am quite new to R statistics, and I one you can help me. I have tried finding the answer to my question by searching the forum and so on, and I apologize in advance if my question is trivial or ...
0
votes
0answers
9 views

Mean of vectors minimizes L2-norm [duplicate]

Recently introduced to data mining and am trying to understand how the mean of vectors minimizes L2-Norm/Euclidean distance. I've tried googling it and can't seem to find a proof as to how something ...
1
vote
3answers
58 views

Finding association rules / frequent Itemsets - what are the application restrictions

What are the restrictions of application fields in searching for association rules (finding frequent itemsets)? All examples I came across cover topic of 'true' basket-analysis in the sense of using ...
0
votes
1answer
34 views

Regression/classification, how to accommodate the missing columns of data?

I would like to apply any regression methods, such as the ones available using WEKA libraries (for example, SVMs, NNs, Random Trees,...) . However, I am getting very low results since I am missing the ...
0
votes
0answers
35 views

Optimally distributing sports tickets to customers

I work for a business that has a certain number of sports tickets to distribute to customers and potential customers every year, and I've been asked to develop a model that will give insight into ...
0
votes
0answers
22 views

Live selection of movie to suggest based on similarity of users

I am working with movie selection for users. 1 ) One of the first ways I thought was taking all the clicked only movie data and building decision trees out of it. Then when input is passed, the ...
0
votes
0answers
14 views

get a list of predicted variables based on rank

I have a data of device, location, age etc of some users, along with keywords that they clicked (or not). Based on this I want to build a model that predicts a list of keywords for test set that ...
3
votes
3answers
127 views

Optimal classification model for translating words

I have the following problem: I have a set of English words which I want to translate to Dutch. Of each words I mined a set of possible translations. For example, for the word "Eighteen" I obtained ...
0
votes
0answers
25 views

Normalize data in unnormalized data after normalization

i have some data in numeric. I want to do some classifications method. So i decided to check the normalization of the data. I have normalize my data using weka tools. And i think weka had normalized ...
0
votes
0answers
20 views

Determine which variable or variables is/are the most efficient to predict the outcome

I have a small dataset (n=74) with a +/- 50 variables, not the best data but I have to work with it. The variables are used to select a product. I want to determine which variable or variables is/are ...
0
votes
2answers
25 views

How to propositionalize a relational data set for clustering analysis?

I am working with a data set of students and their courses for a single semester, attempting to cluster based on the courses & various other attributes where "courses" are the "many" side of a ...
1
vote
1answer
68 views

Linear regression of 0/1 response (Fig. 2.1 of The elements of statistical learning)

In chapter 2 ESL book authors write: Let's look at example of linear model in a classification context They fit a simple linear model $g = 0.3290614 -0.0226360\cdot x_1 + 0.2495983 \cdot x_2 + e$, ...
1
vote
0answers
21 views

Extracting city name from free text?

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names. For example, "Shanghai, China" "China, ShangHai" might mean the ...