Tagged Questions

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

1
vote
1answer
18 views

Why AUC-PR increases when the number of positives increase?

I asked a question earlier about comparing models using Precision-Recall AUC. One of the answers included the following statement: "The larger the fraction of positives in the data set, the larger the ...
1
vote
1answer
19 views

Ranking two models based on ROC-AUC and PR-AUC

I have two methods/classifiers (completely different models) that I need to decide which one is better. The dataset is imbalanced. I trained both classifiers on the same dataset and then I computed ...
2
votes
1answer
20 views

Is it correct to use Precision-Recall AUC in a balanced dataset situation?

I have a binary classification scenario with a dataset that is unbalanced (much more negatives than positives). When I train a classifier on this dataset I get a Precision-Recall AUC of 0.7. Then I ...
0
votes
0answers
14 views

How to calculate the area under the precision-recall curve for the random classifier?

I know that the random classifier score in ROC AUC (Area under the curve) is always 0.5. My question is: how to calculate the Area under the precision-recall curve for the random classifier?
-1
votes
0answers
26 views

Queries in GBM Model in R

I have the following questions pertaining to GBM Model in R. Question 1 : I have a mixture of factorial and numerical data. And Its a binary classification ...
0
votes
0answers
19 views

How to plot a precision-recall curve when doing cross-validation?

I'm using cross-validation to evaluate the performance of a classifier with scikit-learn and I want to plot the Precision-Recall curve. I found an example on ...
4
votes
1answer
71 views

Unintuitive interpretation of probabilities when doing logistic regression

The observations in my dataset can be split in two classes. The observations in class 1 are for sure correctly labeled. The observations that has been designated to class 2 have a huge percentage of ...
0
votes
0answers
24 views

clustering analysis of large amount of time series

I would like to cluster a set of time series, which are composed of around 50000 different time series. Are there established algorithms/package that can handle this scalability problem?
1
vote
0answers
49 views

Incremental aggregation of statistics

I need your suggestion to solve this problem: Given a Data Warehouse with e.g. 10 million entries, additional 1000 entries arrive each day. Rather than recomputing the desired aggregates, an ...
-1
votes
0answers
17 views

Data mining tool for mining e-mail data?

I am very new to Data Mining and need to perform it in on the e-mail data set of an organization in order to extract information from the subject and the body of the message. Note that the objective ...
-1
votes
2answers
41 views

Will RandomForest model work well if the correlation lies between the given attributes

Currently I am working on the data mining project and I am using RandomForest classification model for that. I have a few queries in that. Will the RandomForest handle if there is correlation ...
0
votes
1answer
42 views

What is the convex hull in ROC curve?

I'm reading a paper about ROC and PR curves. They mentioned the ROC convex hull but they don't define it or say what it is. Can someone please tell me the meaning of it? What is a convex hull in ROC ...
2
votes
2answers
44 views

An intuitive meaning of the area under the PR curve?

Wikipedia says that an interpretation of the area under the ROC curve is: "the area under the curve is equal to the probability that a classifier will rank a randomly chosen positive instance higher ...
0
votes
1answer
30 views

build model with complicated types of feature variables

I have been asked to build a model to predict a life span of a material based on a couple of features. The features can be classed into the following categories: 1) The feature variables just have 0 ...
-1
votes
0answers
17 views

Does PCA need to be repeated for data sets with more than 2 dimensions?

Out of all PCA examples I have found, it has always been about finding the eigenvectors and eigenvalues of a data set in two dimensions, as in square matrices, which is 2 dimensional( correct me if ...
4
votes
2answers
132 views

What are some good examples of exploratory data analysis today?

Are there some papers published which illustrate EDA used to tackle substantial data problems? I am particularly looking for actual (current) data examples, where plots have been made and statistics ...
0
votes
0answers
20 views

How to measure distance between two elements in a web page

Suppose we want to measure how closely relevant is one word or phrase to another in a webpage, this can be easily seen by eyes whether they are near. But in an HTML document it could be tricky to ...
0
votes
0answers
10 views

M statistic and software ProtoArray Prospector

I would like to reproduce the statistical result in this paper: http://www.ncbi.nlm.nih.gov/pubmed/24947363 They said "the M-statistic test in ProtoArray Prospector was applied to the normalized ...
2
votes
3answers
111 views

Good sources to get papers in machine learning?

I'm a CS master student (will be a PhD in three months or so). Today I was at my superviser's office and he had a friend discussing some ideas for their research. Then they mentioned a paper that I ...
0
votes
0answers
7 views

Multiple combinations (ex drug-ADR) with the same unique case ID.

I am quite new to R statistics, and I one you can help me. I have tried finding the answer to my question by searching the forum and so on, and I apologize in advance if my question is trivial or ...
0
votes
0answers
9 views

Mean of vectors minimizes L2-norm [duplicate]

Recently introduced to data mining and am trying to understand how the mean of vectors minimizes L2-Norm/Euclidean distance. I've tried googling it and can't seem to find a proof as to how something ...
1
vote
3answers
45 views

Finding association rules / frequent Itemsets - what are the application restrictions

What are the restrictions of application fields in searching for association rules (finding frequent itemsets)? All examples I came across cover topic of 'true' basket-analysis in the sense of using ...
0
votes
1answer
30 views

Regression/classification, how to accommodate the missing columns of data?

I would like to apply any regression methods, such as the ones available using WEKA libraries (for example, SVMs, NNs, Random Trees,...) . However, I am getting very low results since I am missing the ...
0
votes
0answers
35 views

Optimally distributing sports tickets to customers

I work for a business that has a certain number of sports tickets to distribute to customers and potential customers every year, and I've been asked to develop a model that will give insight into ...
0
votes
0answers
22 views

Live selection of movie to suggest based on similarity of users

I am working with movie selection for users. 1 ) One of the first ways I thought was taking all the clicked only movie data and building decision trees out of it. Then when input is passed, the ...
0
votes
0answers
14 views

get a list of predicted variables based on rank

I have a data of device, location, age etc of some users, along with keywords that they clicked (or not). Based on this I want to build a model that predicts a list of keywords for test set that ...
3
votes
3answers
125 views

Optimal classification model for translating words

I have the following problem: I have a set of English words which I want to translate to Dutch. Of each words I mined a set of possible translations. For example, for the word "Eighteen" I obtained ...
0
votes
0answers
23 views

Normalize data in unnormalized data after normalization

i have some data in numeric. I want to do some classifications method. So i decided to check the normalization of the data. I have normalize my data using weka tools. And i think weka had normalized ...
0
votes
0answers
19 views

Determine which variable or variables is/are the most efficient to predict the outcome

I have a small dataset (n=74) with a +/- 50 variables, not the best data but I have to work with it. The variables are used to select a product. I want to determine which variable or variables is/are ...
0
votes
2answers
24 views

How to propositionalize a relational data set for clustering analysis?

I am working with a data set of students and their courses for a single semester, attempting to cluster based on the courses & various other attributes where "courses" are the "many" side of a ...
1
vote
1answer
63 views

Linear regression of 0/1 response (Fig. 2.1 of The elements of statistical learning)

In chapter 2 ESL book authors write: Let's look at example of linear model in a classification context They fit a simple linear model $g = 0.3290614 -0.0226360\cdot x_1 + 0.2495983 \cdot x_2 + e$, ...
1
vote
0answers
18 views

Extracting city name from free text?

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names. For example, "Shanghai, China" "China, ShangHai" might mean the ...
0
votes
2answers
45 views

Instance weighing in libsvm/liblinear

I often use the instance weights with Libsvm for classification problems. http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#weights_for_data_instances Does anyone know the details of the algorithm that ...
0
votes
3answers
43 views

The role of the bias terms in matrix factorization formulas?

I'm reading about matrix factorization for recommender systems. A basic matrix factorization model would be something like: $(p_i \times q_j ) + b_i + b_j$. That formula would compute the rating for ...
0
votes
0answers
25 views

how to handle the imbalanced data in regression analysis

The problem here is very similar to the problem asked by someben in 2012 (link:Sampling for Imbalanced Data in Regression). It involves the linear regression analysis using an unbalanced dataset. Say, ...
0
votes
0answers
26 views

Can sequential pattern mining be called an unsupervised learning method?

Sequential pattern mining finds patterns in unlabeled data. So I assume its an unsupervised learning algorithm. But most references to pattern mining never mention the learning part. Why? Can ...
3
votes
1answer
53 views

R find key influencers

In Excel's data mining tools there is a "Key Influencers" tool which will look at a dataset which is perhaps customers and whether or not they converted to a given goal (e.g. a flag that equals 1). It ...
2
votes
2answers
83 views

ROC for more than 2 outcome categories

How do you construct ROC Curves when there are more than two outcome categories (in my case, I have four)? I've heard you should do this for the most popular group. Are there any other ideas? Are ...
0
votes
1answer
68 views

problems to build the linear multiple regression in R and to explain the output [closed]

I have a problem to build and to explain the linear multiple regression. I have a data set called Cars93 with 26 variables (numeric and not numeric) and 93 ...
2
votes
2answers
32 views

Get the distribution of a dataset

this is a conceptual question. In data mining, the problem often arises that scientists/data engineers are using an expert guess for the underlying distribution of their data. Often their assumption ...
0
votes
0answers
5 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
2
votes
0answers
76 views

How many eligible bachelors in a city?

This is a very simple question posed to me by a friend of mine. I know it's a statistical analysis problem, but I suck at math. Given the total population of $x$ within a metropolitan area, what ...
0
votes
0answers
8 views

automatic assign class name based on text

My question is , I have a set of plain text , i want to create category based on the text. Eg: i have written something about Soup recepie then the algorithm must create a category called Food. After ...
0
votes
0answers
50 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
0
votes
0answers
47 views

How can i make a fraud detection dataset (I have the data ready but unordered)

I'm a little confused with the creation of the dataset for a fraud detection predictive model. Here i put a link with a sample of the dataset that I made. (the real dataset have ~950.000 clients). ...
0
votes
0answers
29 views

Which statistics to use in order to understand a dataset?

So I have a dataset that I will use to train a bunch of classifiers. I need to do that for my thesis. However I'm not sure which statistics are good to use to better understand the dataset and the ...
0
votes
1answer
41 views

Understanding output stepAIC

I am using the stepAIC function in R to do a bi-directional (forward and backward) stepwise regression. I do not understand what each return value from the function ...
2
votes
0answers
26 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
2
votes
2answers
78 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
0
votes
2answers
66 views

Temporal abstraction in Churn analysis: Why do we need it?

Could you explain the need of temporal abstraction in churn analysis intuitively with a simple example? I tried Google but there are not any clear answers , especially for churn analysis.