Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

1
vote
1answer
15 views

Geodesic distance and mean

My data-set consists of points in globe. Suppose a User visits locations $l_1,l_2,\dots l_n$ (each location in $(lat, long)$ in the city with probability $p_1,p_2,\dots,p_n$ and I want to calculate ...
0
votes
0answers
19 views

Anomaly detection of web browsing sequences

Please consider that I'm quite new to machine learning. I need to create models based on browsing patterns of web users and find deviations from that model. I'm using web server access log files. For ...
0
votes
2answers
26 views

Suggestion for discovering inherent patterns in data

I have a a big data set of clients with all sorts of variables that describe their background, payment history, and more... I also have a subset of those client who all have portrayed similar ...
2
votes
2answers
201 views

What statistical techniques to use to distinguish between two groups?

I have a dataset of about 300 people. 200 test positive for a disease, and the rest test negative. I have data on different test scores and imaging results for these 300 participants. So my dataset ...
1
vote
3answers
20 views

Computing Image Similarity based on Color Distribution

Image Similarity based on Color Palette Distribution I am trying to compute similarity between two images based on their color palette distribution, let's say I have two sets of key value pairs as ...
0
votes
1answer
16 views

Spectral clustering using Technics other than kmeans

In spectral clustering, the algorithm suggests performing K-means to k eigenvectors of the resulted Laplacian matrix. My question is: 'Can I use other clustering algorithms such as K-medoids or other ...
0
votes
0answers
15 views

What are some data mining techniques for analyzing cause of disease

I have a dataset of 300 observations, of which 200 are normal, and the rest have the disease. I have the cognitive assessment scores of these 300 participants, and the assessment is divided into ...
0
votes
0answers
12 views

Propensity in linear models and bilinear regression models

I'm reading this paper about matrix factorization. In the paper they want to combine the features of the nodes in the model (page 6). First they illustrate the simple idea of combining the features of ...
0
votes
0answers
16 views

Intuition behind matrix factorization formulations?

I'm reading this paper about matrix factorization. In the paper they propose to use this factorization for the adjacency (or similarity) matrix $G$ using the following formulation: $G = U \Lambda U^T$ ...
-1
votes
0answers
17 views

Association rule mining in R

I have a transaction data from which I am trying to find frequent itemsets and association rules using R . For Example {Milk,Bread}=>{Cheese}. The classical ...
-2
votes
0answers
14 views

calling weka classifier in matlab with cossValidateModel()

i am working on feature selection problem and calling weka classifiers in matlab for checking classification accuracy of selected features. i have called weka classifier with evaluate model ...
0
votes
1answer
44 views

Spatially auto-regressive two-stage model

I'm working on a project in which I use a 'Generalized Spatial Two-Stage Least Squares' model, mostly known as $y= X \beta + \lambda W y + u$ and $u = \rho M u + \epsilon_n$ where $y$ and $u$ are ...
13
votes
7answers
2k views

What is the daily job routine of the machine learning scientist?

I'm a master CS student in a German university now writing my thesis. I will be done in two months I have to make the very hard decision if I should continue with a PhD or find a job in the industry. ...
0
votes
1answer
17 views

Regression-tree Tuning in a Streaming Setting

Some time ago I went through a NIPS 2013 paper Regression-tree Tuning in a Streaming Setting. The paper proposes a tree-based regressor. Is there any implementation of this algorithm available? (At ...
1
vote
2answers
118 views

High precision with low recall SVM

I'm classifying a data set using SVM and those are the precision and recall values for two classes. ...
1
vote
0answers
14 views

Difference between Factorization machines and Matrix Factorization?

I came across the term Factorization Machines in recommender systems. I know what Matrix Factorization is for recommender systems but never heard of Factorization Machines. So what's the difference?
0
votes
1answer
13 views

Assumption behind few latent features in recommender systems?

I know in recommender systems you have a rating matrix and then you factorize this matrix into two matrices and then learn those matrices with gradient descent. In those matrices we specify the number ...
0
votes
0answers
26 views

How to represent bayesian loss function in binary classification

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set ...
1
vote
0answers
20 views

An association rules algorithm that maintains the order of items

For example: if my dataset contains (A, B), but does not contain (B, A). Then the algorithm may generate the rule A -> B, but will not generate the rule B -> A. Is there an association rules ...
2
votes
1answer
48 views

How to solve for the parameters in a logistic function? [duplicate]

I want to find the parameters of a logistic function. I read the guide here. It has a very clear explanation, but it did not have the final solution that I need. Now, we will consider a basis ...
1
vote
0answers
9 views

What does Class Complexity mean in Weka?

When running Weka on my dataset, in the results printout I get the following rows: ...
1
vote
3answers
153 views
+50

How to perform proper data mining on time-series data?

I have some daily data from city A, B, C. Values from city A are highly correlated with values from other cities for lag -1,-2,-3 and -4. I want to use Random Forest, SVM and ANN to predict values ...
0
votes
0answers
12 views

How to identify contributors in groups of people

I have a set of 120,000 people. These people are assigned to 10 million groups. Each of these groups have a target variable indicating the kind of output it made (A, B, C or D). I'd like to identify ...
0
votes
1answer
38 views

Approach for mapping consumer preferences

I have this web application where I need to map consumer preferences based on some input information and individual choices. My goal is to create a list of product recommendations and evaluate the ...
2
votes
2answers
32 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
1
vote
1answer
36 views

Meaning of latent features?

I'm trying to understand matrix factorization models for recommender systems and I always read 'latent features', but what does that mean? I know what a feature means for a training dataset but I'm ...
1
vote
0answers
24 views

In CHAID, shouldn't we merge categories when p<alpha rather than p>alpha?

In CHAID, the categories are merged when P>alpha in the first step. BUT Since CHAID uses Chi-square statisitic, if p-value < alpha, we reject the null ( Independence), hence, meaning the ...
2
votes
1answer
101 views

What loss function can I use for linear classification?

I have a question about loss function in bayes classification. Let see similar case of loss function in linear classification: Given data $(x,y)=${$(x_1,y1)....(x_n,y_n)$} is map to label $T=${-1,1}. ...
0
votes
0answers
14 views

A framework for comparing the performance of Expectation Maximization

I have my own implementation of the Expectation Maximization (EM) algorithm based on the following paper http://pdf.aminer.org/000/221/588/fuzzy_k_means_clustering_with_crisp_regions.pdf I would like ...
0
votes
1answer
64 views

What data mining/machine learning approach to use for a scoring model?

Suppose I have a large data set with lots of features(attributes). And I'm tasked to build some kind of scoring model to rank certain objects with all these features. How do I go about doing this? ...
0
votes
0answers
11 views

SVM Numerical example (step by step)

I have a constant problem understanding SVM for both linear and non-linear separable cases. I understand upto a point that SVM establishes a hyperplane that has the maximum or optimal distance between ...
3
votes
1answer
39 views

When is it appropriate to use PCA as a preprocessing step?

I understand that PCA is used for dimensionality reduction to be able to plot datasets in 2D or 3D. But I have also seen people applying PCA as a preprocessing step in classification scenarios where ...
1
vote
1answer
19 views

Plotting Scatterplot Matrix or Correlation matrix or both?

I have a problem where I want to use a classifier for it. So I defined a set of features and created a dataset. Now I want to generate some plots to understand the features. I came across the ...
0
votes
0answers
27 views

Calculating the variance of a model?

I often hear about the bias-variance tradeoff to evaluate classifiers. Now I want to calculate them. I often compute the AUC of a binary classifier to evaluate its performance and do a 10-fold ...
0
votes
1answer
23 views

Alternatives to absolute error?

Let me explain my scenario in which I need to calculate absolute error. Lets say the X is the actual value. And X' is the value of X with some error 'e'. So X' = X + e'. Lets say i = 1 to 10000. I ...
0
votes
0answers
36 views

Most Important Stat Theory Concepts — Interview [duplicate]

I have an interview with a top company for a data scientist position. I was made aware that they will be testing probability/statistical theory concepts. So the question: If you had 1 hour tops ...
4
votes
2answers
148 views

Ensembles of Ensembles?

I like the idea of ensemble learners, especially Bagging, but I always wondered as why they are not the most powerful learners since they have a clean motivation. I don't have the answer to that ...
2
votes
2answers
50 views

How to get rid of bias in data?

I have been trying to classify a set of data into one of four classes. The data has already been generated and I have set aside 10,000 for training and 2,000 for testing. I have also generated the ...
2
votes
1answer
65 views

The usage of data mining in pharmaceutical companies?

I know that data mining applications are being used in pharmaceutical companies, but my question is: what do they use them for? Sometimes I read: "drug discovery", but how? How is it used for drug ...
0
votes
1answer
57 views

Data Mining, generating a model from a database

I have a database with different variables which contains information such as age, date of vaccination, number of doses as well as the number of antibodies (which is my target variable). If the number ...
0
votes
0answers
38 views

Analyzing neuralnet functions in R

The 'neuralnet' package in R allows us to use neural network algorithm with backpropagation. I want to use the function for prediction. I saw a tutorial on neuralnet in which predictions on the iris ...
1
vote
0answers
12 views

Combine variables that are extremely lightly populated?

A similar question to my other question about mixed distributions. Here i have quite a few variables that are populated to less than 5%, many are even populated to less than 1% this 1% would represent ...
0
votes
1answer
32 views

Data preparation and machine algo for ad click prediction

I am an ml noob. I have a task at hand of predicting click probability given user information like city, state, os version, os family, device, browser family browser version, city, etc. I have been ...
0
votes
1answer
14 views

How to increase a particular terms's weightage?

I am doing Text classification using LibSVM in Rapid Miner. I am using TFIDF values for processing documents. I need to Increase weightage of some terms in the documents(for eg. words in BOLD and ...
0
votes
0answers
11 views

Data Mining technique for preferred vendor problem?

Currently I use open auction method for selecting a transporter and want to switch to preferred transporter where I would select a transporter based on my preferred transporter list. The project ...
4
votes
2answers
283 views

What is the difference between a loss function and decision function

I see that both functions are part of data mining methods such as Gradient Boosting Regressors. I see that those are seperate objects. But how is the relationship between both in general?
1
vote
0answers
21 views

How to do the prediction with text mining and rating score?

I have the data as following: rating document1 document2 5 some words some words 3 some words some words 2 some words some words ...
4
votes
3answers
269 views

The idea of making the data have a zero-mean

I often see people making a dimension/feature of a dataset be of a zero-mean by removing the mean from all the elements. But I never I understood why to do so? What is the effect of doing that as a ...
3
votes
1answer
58 views

Inputs to k-means are often normalized per-feature. Why not fully whiten the data instead?

We often normalize inputs to the k-means algorithm by 1) subtracting the mean on a per-feature basis and 2) dividing by the standard deviation on a per-feature basis. Some rational behind this is ...
0
votes
1answer
31 views

Best practice to understand a dataset

My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct. My questions: What are ...