Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

0
votes
1answer
9 views

Assumption behind few latent features in recommender systems?

I know in recommender systems you have a rating matrix and then you factorize this matrix into two matrices and then learn those matrices with gradient descent. In those matrices we specify the number ...
0
votes
0answers
22 views

How to represent bayesian loss function in binary classification

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set ...
-4
votes
0answers
12 views

weka classifiers with cross validation implementation in matlab [on hold]

i want to call weka 3.7 cross validate model in matlab? can any one provide me code in matlab or tell me what exact parameters should be passed to crossValidateModel???
1
vote
0answers
12 views

An association rules algorithm that maintains the order of items

For example: if my dataset contains (A, B), but does not contain (B, A). Then the algorithm may generate the rule A -> B, but will not generate the rule B -> A. Is there an association rules ...
2
votes
1answer
42 views

How to solve for the parameters in a logistic function? [duplicate]

I want to find the parameters of a logistic function. I read the guide here. It has a very clear explanation, but it did not have the final solution that I need. Now, we will consider a basis ...
1
vote
0answers
9 views

What does Class Complexity mean in Weka?

When running Weka on my dataset, in the results printout I get the following rows: ...
0
votes
0answers
33 views

How to perform proper data mining on time-series data?

I have some daily data from city A, B, C. Values from city A are highly correlated with values from other cities for lag -1,-2,-3 and -4. I want to use Random Forest, SVM and ANN to predict values ...
0
votes
0answers
9 views

How to identify contributors in groups of people

I have a set of 120,000 people. These people are assigned to 10 million groups. Each of these groups have a target variable indicating the kind of output it made (A, B, C or D). I'd like to identify ...
0
votes
1answer
27 views

Approach for mapping consumer preferences

I have this web application where I need to map consumer preferences based on some input information and individual choices. My goal is to create a list of product recommendations and evaluate the ...
2
votes
2answers
32 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
1
vote
1answer
36 views

Meaning of latent features?

I'm trying to understand matrix factorization models for recommender systems and I always read 'latent features', but what does that mean? I know what a feature means for a training dataset but I'm ...
1
vote
0answers
23 views

In CHAID, shouldn't we merge categories when p<alpha rather than p>alpha?

In CHAID, the categories are merged when P>alpha in the first step. BUT Since CHAID uses Chi-square statisitic, if p-value < alpha, we reject the null ( Independence), hence, meaning the ...
2
votes
1answer
98 views

What loss function can I use for linear classification?

I have a question about loss function in bayes classification. Let see similar case of loss function in linear classification: Given data $(x,y)=${$(x_1,y1)....(x_n,y_n)$} is map to label $T=${-1,1}. ...
0
votes
0answers
14 views

A framework for comparing the performance of Expectation Maximization

I have my own implementation of the Expectation Maximization (EM) algorithm based on the following paper http://pdf.aminer.org/000/221/588/fuzzy_k_means_clustering_with_crisp_regions.pdf I would like ...
0
votes
1answer
58 views

What data mining/machine learning approach to use for a scoring model?

Suppose I have a large data set with lots of features(attributes). And I'm tasked to build some kind of scoring model to rank certain objects with all these features. How do I go about doing this? ...
0
votes
0answers
11 views

SVM Numerical example (step by step)

I have a constant problem understanding SVM for both linear and non-linear separable cases. I understand upto a point that SVM establishes a hyperplane that has the maximum or optimal distance between ...
3
votes
1answer
36 views

When is it appropriate to use PCA as a preprocessing step?

I understand that PCA is used for dimensionality reduction to be able to plot datasets in 2D or 3D. But I have also seen people applying PCA as a preprocessing step in classification scenarios where ...
1
vote
1answer
18 views

Plotting Scatterplot Matrix or Correlation matrix or both?

I have a problem where I want to use a classifier for it. So I defined a set of features and created a dataset. Now I want to generate some plots to understand the features. I came across the ...
0
votes
0answers
27 views

Calculating the variance of a model?

I often hear about the bias-variance tradeoff to evaluate classifiers. Now I want to calculate them. I often compute the AUC of a binary classifier to evaluate its performance and do a 10-fold ...
0
votes
1answer
22 views

Alternatives to absolute error?

Let me explain my scenario in which I need to calculate absolute error. Lets say the X is the actual value. And X' is the value of X with some error 'e'. So X' = X + e'. Lets say i = 1 to 10000. I ...
0
votes
0answers
35 views

Most Important Stat Theory Concepts — Interview [duplicate]

I have an interview with a top company for a data scientist position. I was made aware that they will be testing probability/statistical theory concepts. So the question: If you had 1 hour tops ...
4
votes
2answers
142 views

Ensembles of Ensembles?

I like the idea of ensemble learners, especially Bagging, but I always wondered as why they are not the most powerful learners since they have a clean motivation. I don't have the answer to that ...
2
votes
2answers
48 views

How to get rid of bias in data?

I have been trying to classify a set of data into one of four classes. The data has already been generated and I have set aside 10,000 for training and 2,000 for testing. I have also generated the ...
2
votes
1answer
59 views

The usage of data mining in pharmaceutical companies?

I know that data mining applications are being used in pharmaceutical companies, but my question is: what do they use them for? Sometimes I read: "drug discovery", but how? How is it used for drug ...
0
votes
1answer
53 views

Data Mining, generating a model from a database

I have a database with different variables which contains information such as age, date of vaccination, number of doses as well as the number of antibodies (which is my target variable). If the number ...
0
votes
0answers
36 views

Analyzing neuralnet functions in R

The 'neuralnet' package in R allows us to use neural network algorithm with backpropagation. I want to use the function for prediction. I saw a tutorial on neuralnet in which predictions on the iris ...
1
vote
0answers
12 views

Combine variables that are extremely lightly populated?

A similar question to my other question about mixed distributions. Here i have quite a few variables that are populated to less than 5%, many are even populated to less than 1% this 1% would represent ...
0
votes
1answer
28 views

Data preparation and machine algo for ad click prediction

I am an ml noob. I have a task at hand of predicting click probability given user information like city, state, os version, os family, device, browser family browser version, city, etc. I have been ...
0
votes
1answer
13 views

How to increase a particular terms's weightage?

I am doing Text classification using LibSVM in Rapid Miner. I am using TFIDF values for processing documents. I need to Increase weightage of some terms in the documents(for eg. words in BOLD and ...
0
votes
0answers
11 views

Data Mining technique for preferred vendor problem?

Currently I use open auction method for selecting a transporter and want to switch to preferred transporter where I would select a transporter based on my preferred transporter list. The project ...
4
votes
2answers
281 views

What is the difference between a loss function and decision function

I see that both functions are part of data mining methods such as Gradient Boosting Regressors. I see that those are seperate objects. But how is the relationship between both in general?
1
vote
0answers
19 views

How to do the prediction with text mining and rating score?

I have the data as following: rating document1 document2 5 some words some words 3 some words some words 2 some words some words ...
4
votes
3answers
262 views

The idea of making the data have a zero-mean

I often see people making a dimension/feature of a dataset be of a zero-mean by removing the mean from all the elements. But I never I understood why to do so? What is the effect of doing that as a ...
3
votes
1answer
56 views

Inputs to k-means are often normalized per-feature. Why not fully whiten the data instead?

We often normalize inputs to the k-means algorithm by 1) subtracting the mean on a per-feature basis and 2) dividing by the standard deviation on a per-feature basis. Some rational behind this is ...
0
votes
1answer
30 views

Best practice to understand a dataset

My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct. My questions: What are ...
42
votes
7answers
6k views

Skills hard to find in machine learners?

It seems that data mining and machine learning became so popular that now almost every CS student knows about classifiers, clustering, statistical NLP ... etc. So it seems that finding data miners is ...
2
votes
3answers
59 views

Matrix Factorization algorithms for Recommender Systems

I need to learn about Matrix Factorization for recommender systems, so I downloaded this paper https://datajobs.com/data-science-repo/Recommender-Systems-[Netflix].pdf but I found it too shallow. It ...
0
votes
0answers
13 views

Cross validation to test the performance of two spatial interpolation methods

I have 14 weather stations' temperature data for the period between 2010–2013. I need to evaluate the performance of two spatial interpolation methods. I suggest to select 10 days from this period of ...
2
votes
2answers
51 views

Are there any good publicly accessible wisdom of crowd data sets?

I am looking for publicly accessible data sets to explore some wisdom of the crowd ideas. An ideal data set would contain predictions from a number of individuals (who were not collaborating with ...
1
vote
1answer
35 views

What is the difference between a parametric test and non-parametric tests? [duplicate]

Please explain the difference between a parametric and a non-parametric test. Do all data mining techniques come under the non-parametric category?
0
votes
0answers
21 views

Predictions on statistical series

I need to summarize possible approaches (as well as their pros and cons) to the problem of statistical series prediction. It's pure theoretical question, an example below was given only for ...
0
votes
0answers
11 views

Finding Repeating Combinations

I have a table where Field1 = Name {John, Mark, Brian} and Field2 = Ball Color {Red, Orange, Yellow} Each person has 6 balls of different colors so for example ...
1
vote
1answer
41 views

Different hierarchical clustering results

I'm running a hierarchical clustering on a sample of data using the steps below: ...
0
votes
0answers
30 views

What are the mathematics I need to learn, before I start research in data mining [duplicate]

I usually use text mining, graph mining, Information retrieval, and natural lanuage processing. Also i will use the fundamental concepts of data mining like classification, association and clustering. ...
1
vote
0answers
56 views

What should I use - Multi label classification or Multi class classification?

In my dataset, I have 2 labels, positive and negative. Most samples belong to only one class, either positive or negative. A small fraction of samples take both labels i.e. both positive and negative. ...
0
votes
0answers
20 views

Association Rules “with a kind of class”

I want to use/adapt a recommendation algorithm for posters in an e-commerce. The thing is that I want to use previous categories searched before posting in a particular category (has to be at a very ...
-2
votes
2answers
142 views

questions that can be asked for linear regression and logistic regression

I am preparing a potential interview, which can involve detailed questions on linear regression and logistic regression. Besides going over the textbook, could you share some possible interview ...
0
votes
0answers
64 views

SVD application for a Boolean sparse Matrix

Basically, I am trying to have a recommender system based on SVD for a Boolean utility matrix. ie If at all some entries are present in the utility matrix, they will be 1 (I made it pseudo-implicit ...
0
votes
1answer
62 views

Question on Machine Learning Overview?

I am a researcher from India and working in the field of Computational Linguistics for quite some time. I have lately started working in the field of Machine Learning based algorithms. To do this I ...
0
votes
0answers
32 views

Examples for commercial applications of data mining in pharmacy

I've always heard that data mining and machine learning tools/techniques are heavily used in the pharmacy sector and biology but I have never heard of companies that offer commercial applications of ...