Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

1
vote
0answers
11 views

Optimal classification model for translating words

I have the following problem: I have a set of English words which I want to translate to Dutch. Of each words I mined a set of possible translations. For example, for the word "Eighteen" I obtained ...
0
votes
0answers
17 views

Normalize data in unnormalized data after normalization

i have some data in numeric. I want to do some classifications method. So i decided to check the normalization of the data. I have normalize my data using weka tools. And i think weka had normalized ...
0
votes
0answers
17 views

Determine which variable or variables is/are the most efficient to predict the outcome

I have a small dataset (n=74) with a +/- 50 variables, not the best data but I have to work with it. The variables are used to select a product. I want to determine which variable or variables is/are ...
0
votes
0answers
8 views

Script for downloading Medline abstracts with PMID [migrated]

I am looking for a script that will enable me to download a set of abstracts (or article meta-data of an article) from PubMed by supplying it with a list of PubMed ID (PMID) numbers (e.g., from a .csv ...
0
votes
2answers
23 views

How to propositionalize a relational data set for clustering analysis?

I am working with a data set of students and their courses for a single semester, attempting to cluster based on the courses & various other attributes where "courses" are the "many" side of a ...
1
vote
1answer
53 views

Linear regression of 0/1 response (Fig. 2.1 of The elements of statistical learning)

In chapter 2 ESL book authors write: Let's look at example of linear model in a classification context They fit a simple linear model $g = 0.3290614 -0.0226360\cdot x_1 + 0.2495983 \cdot x_2 + e$, ...
1
vote
0answers
15 views

Extracting city name from free text?

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names. For example, "Shanghai, China" "China, ShangHai" might mean the ...
0
votes
2answers
28 views

Instance weighing in libsvm/liblinear

I often use the instance weights with Libsvm for classification problems. http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/#weights_for_data_instances Does anyone know the details of the algorithm that ...
0
votes
2answers
30 views

The role of the bias terms in matrix factorization formulas?

I'm reading about matrix factorization for recommender systems. A basic matrix factorization model would be something like: $(p_i \times q_j ) + b_i + b_j$. That formula would compute the rating for ...
0
votes
0answers
21 views

how to handle the imbalanced data in regression analysis

The problem here is very similar to the problem asked by someben in 2012 (link:Sampling for Imbalanced Data in Regression). It involves the linear regression analysis using an unbalanced dataset. Say, ...
0
votes
0answers
22 views

Can sequential pattern mining be called an unsupervised learning method?

Sequential pattern mining finds patterns in unlabeled data. So I assume its an unsupervised learning algorithm. But most references to pattern mining never mention the learning part. Why? Can ...
3
votes
1answer
43 views

R find key influencers

In Excel's data mining tools there is a "Key Influencers" tool which will look at a dataset which is perhaps customers and whether or not they converted to a given goal (e.g. a flag that equals 1). It ...
2
votes
2answers
77 views

ROC for more than 2 outcome categories

How do you construct ROC Curves when there are more than two outcome categories (in my case, I have four)? I've heard you should do this for the most popular group. Are there any other ideas? Are ...
0
votes
1answer
51 views

problems to build the linear multiple regression in R and to explain the output [closed]

I have a problem to build and to explain the linear multiple regression. I have a data set called Cars93 with 26 variables (numeric and not numeric) and 93 ...
0
votes
0answers
32 views

different feature types for classification

There has a data set with several features. One feature is of the type of continuous numerical values; another feature is of the type of categorical values, such as A, B and C. If I want to build a ...
2
votes
2answers
31 views

Get the distribution of a dataset

this is a conceptual question. In data mining, the problem often arises that scientists/data engineers are using an expert guess for the underlying distribution of their data. Often their assumption ...
0
votes
0answers
5 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
2
votes
0answers
72 views

How many eligible bachelors in a city?

This is a very simple question posed to me by a friend of mine. I know it's a statistical analysis problem, but I suck at math. Given the total population of $x$ within a metropolitan area, what ...
0
votes
0answers
8 views

automatic assign class name based on text

My question is , I have a set of plain text , i want to create category based on the text. Eg: i have written something about Soup recepie then the algorithm must create a category called Food. After ...
0
votes
0answers
39 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
0
votes
0answers
34 views

How can i make a fraud detection dataset (I have the data ready but unordered)

I'm a little confused with the creation of the dataset for a fraud detection predictive model. Here i put a link with a sample of the dataset that I made. (the real dataset have ~950.000 clients). ...
0
votes
0answers
29 views

Which statistics to use in order to understand a dataset?

So I have a dataset that I will use to train a bunch of classifiers. I need to do that for my thesis. However I'm not sure which statistics are good to use to better understand the dataset and the ...
0
votes
1answer
32 views

Understanding output stepAIC

I am using the stepAIC function in R to do a bi-directional (forward and backward) stepwise regression. I do not understand what each return value from the function ...
2
votes
0answers
25 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
2
votes
2answers
78 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
0
votes
2answers
54 views

Temporal abstraction in Churn analysis: Why do we need it?

Could you explain the need of temporal abstraction in churn analysis intuitively with a simple example? I tried Google but there are not any clear answers , especially for churn analysis.
0
votes
1answer
67 views

Calculating the information gain on the features with python

I'm looking for a python library that computes the information gain for the features given a training matrix. Are you aware of any?
0
votes
0answers
24 views

semisupervised classification training on all or part of unlabeled data

I have 3 sets of data. A positively labeled dataset. An unlabeled dataset that has for sure positive (around 75%) and negative data. An unlabeled dataset that has for sure positive data and maybe ...
0
votes
0answers
14 views

Find relations without confounding variables? [duplicate]

I have multiple numerical and categorical variables which I'd like to data-mine for simple relations. I'd create simple plots of two variables which are supposed to have a meaningful statement. Can ...
0
votes
0answers
8 views

S and N parameters of Ridor

I am using WEKA and in particular their implementation of Ridor. The documentation says this about the parameters S and N: -S Set number of shuffles to randomize the data in order to get better ...
1
vote
0answers
24 views

Low pass filter to maintain edge information

I am looking for a kernel as low pass filter that satisfy as:I must find a kernel that statisfies as follows: In the my reference paper, the author suggest gaussian kernel that is The gaussian ...
0
votes
0answers
9 views

Represents istances with multiple values for an attribute and similarity between them

In the scenario in which I'm working each entity could be represented in terms of ten distinct properties that I will call p1, p2, ..., pn. For each of them, an entity, can have its specific range of ...
1
vote
1answer
33 views

Geodesic distance and mean

My data-set consists of points in globe. Suppose a User visits locations $l_1,l_2,\dots l_n$ (each location in $(lat, long)$ in the city with probability $p_1,p_2,\dots,p_n$ and I want to calculate ...
0
votes
0answers
32 views

Anomaly detection of web browsing sequences

Please consider that I'm quite new to machine learning. I need to create models based on browsing patterns of web users and find deviations from that model. I'm using web server access log files. For ...
0
votes
2answers
29 views

Suggestion for discovering inherent patterns in data

I have a a big data set of clients with all sorts of variables that describe their background, payment history, and more... I also have a subset of those client who all have portrayed similar ...
3
votes
2answers
215 views

What statistical techniques to use to distinguish between two groups?

I have a dataset of about 300 people. 200 test positive for a disease, and the rest test negative. I have data on different test scores and imaging results for these 300 participants. So my dataset ...
1
vote
3answers
30 views

Computing Image Similarity based on Color Distribution

Image Similarity based on Color Palette Distribution I am trying to compute similarity between two images based on their color palette distribution, let's say I have two sets of key value pairs as ...
0
votes
1answer
18 views

Spectral clustering using Technics other than kmeans

In spectral clustering, the algorithm suggests performing K-means to k eigenvectors of the resulted Laplacian matrix. My question is: 'Can I use other clustering algorithms such as K-medoids or other ...
0
votes
1answer
22 views

What are some data mining techniques for analyzing cause of disease

I have a dataset of 300 observations, of which 200 are normal, and the rest have the disease. I have the cognitive assessment scores of these 300 participants, and the assessment is divided into ...
0
votes
0answers
20 views

Propensity in linear models and bilinear regression models

I'm reading this paper about matrix factorization. In the paper they want to combine the features of the nodes in the model (page 6). First they illustrate the simple idea of combining the features of ...
1
vote
1answer
32 views

Intuition behind matrix factorization formulations?

I'm reading this paper about matrix factorization. In the paper they propose to use this factorization for the adjacency (or similarity) matrix $G$ using the following formulation: $G = U \Lambda U^T$ ...
0
votes
1answer
50 views

Spatially auto-regressive two-stage model

I'm working on a project in which I use a 'Generalized Spatial Two-Stage Least Squares' model, mostly known as $y= X \beta + \lambda W y + u$ and $u = \rho M u + \epsilon_n$ where $y$ and $u$ are ...
14
votes
7answers
2k views

What is the daily job routine of the machine learning scientist?

I'm a master CS student in a German university now writing my thesis. I will be done in two months I have to make the very hard decision if I should continue with a PhD or find a job in the industry. ...
1
vote
2answers
154 views

High precision with low recall SVM

I'm classifying a data set using SVM and those are the precision and recall values for two classes. ...
1
vote
0answers
34 views

Difference between Factorization machines and Matrix Factorization?

I came across the term Factorization Machines in recommender systems. I know what Matrix Factorization is for recommender systems but never heard of Factorization Machines. So what's the difference?
0
votes
1answer
17 views

Assumption behind few latent features in recommender systems?

I know in recommender systems you have a rating matrix and then you factorize this matrix into two matrices and then learn those matrices with gradient descent. In those matrices we specify the number ...
0
votes
0answers
27 views

How to represent bayesian loss function in binary classification

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set ...
1
vote
0answers
32 views

An association rules algorithm that maintains the order of items

For example: if my dataset contains (A, B), but does not contain (B, A). Then the algorithm may generate the rule A -> B, but will not generate the rule B -> A. Is there an association rules ...
2
votes
1answer
62 views

How to solve for the parameters in a logistic function? [duplicate]

I want to find the parameters of a logistic function. I read the guide here. It has a very clear explanation, but it did not have the final solution that I need. Now, we will consider a basis ...
1
vote
0answers
21 views

What does Class Complexity mean in Weka?

When running Weka on my dataset, in the results printout I get the following rows: ...