Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

1
vote
1answer
41 views

Linear regression of 0/1 response (Fig. 2.1 of The elements of statistical learning)

In chapter 2 ESL book authors write: Let's look at example of linear model in a classification context They fit a simple linear model $g = 0.3290614 -0.0226360\cdot x_1 + 0.2495983 \cdot x_2 + e$, ...
1
vote
0answers
14 views

Extracting city name from free text?

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names. For example, "Shanghai, China" "China, ShangHai" might mean the ...
0
votes
1answer
16 views

Instance weighing in libsvm/liblinear

I often use the instance weights with Libsvm for classification problems. http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#weights_for_data_instances Does anyone know the details of the algorithm that ...
0
votes
2answers
24 views

The role of the bias terms in matrix factorization formulas?

I'm reading about matrix factorization for recommender systems. A basic matrix factorization model would be something like: $(p_i \times q_j ) + b_i + b_j$. That formula would compute the rating for ...
0
votes
0answers
19 views

how to handle the imbalanced data in regression analysis

The problem here is very similar to the problem asked by someben in 2012 (link:Sampling for Imbalanced Data in Regression). It involves the linear regression analysis using an unbalanced dataset. Say, ...
0
votes
0answers
19 views

Can sequential pattern mining be called an unsupervised learning method?

Sequential pattern mining finds patterns in unlabeled data. So I assume its an unsupervised learning algorithm. But most references to pattern mining never mention the learning part. Why? Can ...
0
votes
0answers
12 views

based on the classifiers [closed]

My project is based on filter unwanted messages from osn user walls.In this paper they have used machine learning soft classifier. So,i need the better classifier then this.so can any one suggest me ...
3
votes
1answer
42 views

R find key influencers

In Excel's data mining tools there is a "Key Influencers" tool which will look at a dataset which is perhaps customers and whether or not they converted to a given goal (e.g. a flag that equals 1). It ...
2
votes
2answers
68 views

ROC for more than 2 outcome categories

How do you construct ROC Curves when there are more than two outcome categories (in my case, I have four)? I've heard you should do this for the most popular group. Are there any other ideas? Are ...
0
votes
1answer
46 views

problems to build the linear multiple regression in R and to explain the output [closed]

I have a problem to build and to explain the linear multiple regression. I have a data set called Cars93 with 26 variables (numeric and not numeric) and 93 ...
0
votes
0answers
26 views

different feature types for classification

There has a data set with several features. One feature is of the type of continuous numerical values; another feature is of the type of categorical values, such as A, B and C. If I want to build a ...
2
votes
2answers
29 views

Get the distribution of a dataset

this is a conceptual question. In data mining, the problem often arises that scientists/data engineers are using an expert guess for the underlying distribution of their data. Often their assumption ...
0
votes
0answers
4 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
2
votes
0answers
72 views

How many eligible bachelors in a city?

This is a very simple question posed to me by a friend of mine. I know it's a statistical analysis problem, but I suck at math. Given the total population of $x$ within a metropolitan area, what ...
0
votes
0answers
8 views

automatic assign class name based on text

My question is , I have a set of plain text , i want to create category based on the text. Eg: i have written something about Soup recepie then the algorithm must create a category called Food. After ...
0
votes
0answers
36 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
0
votes
0answers
26 views

How can i make a fraud detection dataset (I have the data ready but unordered)

I'm a little confused with the creation of the dataset for a fraud detection predictive model. Here i put a link with a sample of the dataset that I made. (the real dataset have ~950.000 clients). ...
0
votes
0answers
28 views

Which statistics to use in order to understand a dataset?

So I have a dataset that I will use to train a bunch of classifiers. I need to do that for my thesis. However I'm not sure which statistics are good to use to better understand the dataset and the ...
0
votes
1answer
31 views

Understanding output stepAIC

I am using the stepAIC function in R to do a bi-directional (forward and backward) stepwise regression. I do not understand what each return value from the function ...
2
votes
0answers
22 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
2
votes
2answers
77 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
-2
votes
0answers
15 views

What are the applications of multi-agent approach for large scale data mining

I understand that some large scale data mining can be done via grid computing. So I am wondering if there is any advantage of employing multi-agent approach in this area. Any references or resources? ...
0
votes
2answers
48 views

Temporal abstraction in Churn analysis: Why do we need it?

Could you explain the need of temporal abstraction in churn analysis intuitively with a simple example? I tried Google but there are not any clear answers , especially for churn analysis.
0
votes
1answer
29 views

Calculating the information gain on the features with python

I'm looking for a python library that computes the information gain for the features given a training matrix. Are you aware of any?
0
votes
0answers
21 views

semisupervised classification training on all or part of unlabeled data

I have 3 sets of data. A positively labeled dataset. An unlabeled dataset that has for sure positive (around 75%) and negative data. An unlabeled dataset that has for sure positive data and maybe ...
0
votes
0answers
14 views

Find relations without confounding variables? [duplicate]

I have multiple numerical and categorical variables which I'd like to data-mine for simple relations. I'd create simple plots of two variables which are supposed to have a meaningful statement. Can ...
0
votes
0answers
8 views

S and N parameters of Ridor

I am using WEKA and in particular their implementation of Ridor. The documentation says this about the parameters S and N: -S Set number of shuffles to randomize the data in order to get better ...
1
vote
0answers
24 views

Low pass filter to maintain edge information

I am looking for a kernel as low pass filter that satisfy as:I must find a kernel that statisfies as follows: In the my reference paper, the author suggest gaussian kernel that is The gaussian ...
0
votes
0answers
9 views

Represents istances with multiple values for an attribute and similarity between them

In the scenario in which I'm working each entity could be represented in terms of ten distinct properties that I will call p1, p2, ..., pn. For each of them, an entity, can have its specific range of ...
1
vote
1answer
32 views

Geodesic distance and mean

My data-set consists of points in globe. Suppose a User visits locations $l_1,l_2,\dots l_n$ (each location in $(lat, long)$ in the city with probability $p_1,p_2,\dots,p_n$ and I want to calculate ...
0
votes
0answers
30 views

Anomaly detection of web browsing sequences

Please consider that I'm quite new to machine learning. I need to create models based on browsing patterns of web users and find deviations from that model. I'm using web server access log files. For ...
0
votes
2answers
29 views

Suggestion for discovering inherent patterns in data

I have a a big data set of clients with all sorts of variables that describe their background, payment history, and more... I also have a subset of those client who all have portrayed similar ...
3
votes
2answers
214 views

What statistical techniques to use to distinguish between two groups?

I have a dataset of about 300 people. 200 test positive for a disease, and the rest test negative. I have data on different test scores and imaging results for these 300 participants. So my dataset ...
1
vote
3answers
28 views

Computing Image Similarity based on Color Distribution

Image Similarity based on Color Palette Distribution I am trying to compute similarity between two images based on their color palette distribution, let's say I have two sets of key value pairs as ...
0
votes
1answer
17 views

Spectral clustering using Technics other than kmeans

In spectral clustering, the algorithm suggests performing K-means to k eigenvectors of the resulted Laplacian matrix. My question is: 'Can I use other clustering algorithms such as K-medoids or other ...
0
votes
1answer
20 views

What are some data mining techniques for analyzing cause of disease

I have a dataset of 300 observations, of which 200 are normal, and the rest have the disease. I have the cognitive assessment scores of these 300 participants, and the assessment is divided into ...
0
votes
0answers
17 views

Propensity in linear models and bilinear regression models

I'm reading this paper about matrix factorization. In the paper they want to combine the features of the nodes in the model (page 6). First they illustrate the simple idea of combining the features of ...
0
votes
0answers
18 views

Intuition behind matrix factorization formulations?

I'm reading this paper about matrix factorization. In the paper they propose to use this factorization for the adjacency (or similarity) matrix $G$ using the following formulation: $G = U \Lambda U^T$ ...
0
votes
1answer
48 views

Spatially auto-regressive two-stage model

I'm working on a project in which I use a 'Generalized Spatial Two-Stage Least Squares' model, mostly known as $y= X \beta + \lambda W y + u$ and $u = \rho M u + \epsilon_n$ where $y$ and $u$ are ...
14
votes
7answers
2k views

What is the daily job routine of the machine learning scientist?

I'm a master CS student in a German university now writing my thesis. I will be done in two months I have to make the very hard decision if I should continue with a PhD or find a job in the industry. ...
1
vote
2answers
142 views

High precision with low recall SVM

I'm classifying a data set using SVM and those are the precision and recall values for two classes. ...
1
vote
0answers
29 views

Difference between Factorization machines and Matrix Factorization?

I came across the term Factorization Machines in recommender systems. I know what Matrix Factorization is for recommender systems but never heard of Factorization Machines. So what's the difference?
0
votes
1answer
17 views

Assumption behind few latent features in recommender systems?

I know in recommender systems you have a rating matrix and then you factorize this matrix into two matrices and then learn those matrices with gradient descent. In those matrices we specify the number ...
0
votes
0answers
27 views

How to represent bayesian loss function in binary classification

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set ...
1
vote
0answers
32 views

An association rules algorithm that maintains the order of items

For example: if my dataset contains (A, B), but does not contain (B, A). Then the algorithm may generate the rule A -> B, but will not generate the rule B -> A. Is there an association rules ...
2
votes
1answer
57 views

How to solve for the parameters in a logistic function? [duplicate]

I want to find the parameters of a logistic function. I read the guide here. It has a very clear explanation, but it did not have the final solution that I need. Now, we will consider a basis ...
1
vote
0answers
18 views

What does Class Complexity mean in Weka?

When running Weka on my dataset, in the results printout I get the following rows: ...
1
vote
4answers
292 views

How to perform proper data mining on time-series data?

I have some daily data from city A, B, C. Values from city A are highly correlated with values from other cities for lag -1,-2,-3 and -4. I want to use Random Forest, SVM and ANN to predict values ...
0
votes
0answers
13 views

How to identify contributors in groups of people

I have a set of 120,000 people. These people are assigned to 10 million groups. Each of these groups have a target variable indicating the kind of output it made (A, B, C or D). I'd like to identify ...
0
votes
1answer
41 views

Approach for mapping consumer preferences

I have this web application where I need to map consumer preferences based on some input information and individual choices. My goal is to create a list of product recommendations and evaluate the ...