Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
408
votes
5answers
145k views

How to understand the drawbacks of K-means

K-means is a widely used method in cluster analysis. In my understanding, this method does NOT require ANY assumptions, i.e., give me a dataset and a pre-specified number of clusters, k, and I just ...
218
votes
13answers
200k views

What is the difference between data mining, statistics, machine learning and AI?

What is the difference between data mining, statistics, machine learning and AI? Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different ...
164
votes
4answers
140k views

Cohen's kappa in plain English

I am reading a data mining book and it mentioned the Kappa statistic as a means for evaluating the prediction performance of classifiers. However, I just can't understand this. I also checked ...
139
votes
9answers
55k views

Obtaining knowledge from a random forest

Random forests are considered to be black boxes, but recently I was thinking what knowledge can be obtained from a random forest? The most obvious thing is the importance of the variables, in the ...
84
votes
7answers
33k views

Euclidean distance is usually not good for sparse data (and more general case)?

I have seen somewhere that classical distances (like Euclidean distance) become weakly discriminant when we have multidimensional and sparse data. Why? Do you have an example of two sparse data ...
77
votes
9answers
13k views

Skills hard to find in machine learners?

It seems that data mining and machine learning became so popular that now almost every CS student knows about classifiers, clustering, statistical NLP ... etc. So it seems that finding data miners is ...
74
votes
11answers
44k views

Having a job in data-mining without a PhD

I've been very interested in data-mining and machine-learning for a while, partly because I majored in that area at school, but also because I am truly much more excited trying to solve problems that ...
65
votes
2answers
91k views

Performance metrics to evaluate unsupervised learning

With respect to the unsupervised learning (like clustering), are there any metrics to evaluate performance?
63
votes
2answers
34k views

Why only three partitions? (training, validation, test)

When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset. This is because the models usually have ...
59
votes
12answers
19k views

Software needed to scrape data from graph [closed]

Anybody have any experience with software (preferably free, preferably open source) that will take an image of data plotted on cartesian coordinates (a standard, everyday plot) and extract the ...
57
votes
3answers
55k views

Clustering with K-Means and EM: how are they related?

I have studied algorithms for clustering data (unsupervised learning): EM, and k-means. I keep reading the following : k-means is a variant of EM, with the assumptions that clusters are ...
55
votes
8answers
12k views

Is sampling relevant in the time of 'big data'?

Or more so "will it be"? Big Data makes statistics and relevant knowledge all the more important but seems to underplay Sampling Theory. I've seen this hype around 'Big Data' and can't help wonder ...
54
votes
3answers
2k views

Do we have a problem of “pity upvotes”?

I know, this may sound like it is off-topic, but hear me out. At Stack Overflow and here we get votes on posts, this is all stored in a tabular form. E.g.: post id voter id vote type ...
44
votes
5answers
70k views

Lift measure in data mining

I searched many websites to know what exactly lift will do? The results that I found all were about using it in applications not itself. I know about the support and confidence function. From ...
44
votes
3answers
40k views

What are the differences between hidden Markov models and neural networks?

I'm just getting my feet wet in statistics so I'm sorry if this question does not make sense. I have used Markov models to predict hidden states (unfair casinos, dice rolls, etc.) and neural networks ...
44
votes
2answers
241k views

How to interpret the output of the summary method for an lm object in R? [duplicate]

I am using sample algae data to understand data mining a bit more. I have used the following commands: ...
40
votes
2answers
6k views

How to draw valid conclusions from “big data”?

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
39
votes
1answer
37k views

Relative variable importance for Boosting

I'm looking for an explanation of how relative variable importance is computed in Gradient Boosted Trees that is not overly general/simplistic like: The measures are based on the number of times a ...
35
votes
5answers
5k views

Think like a bayesian, check like a frequentist: What does that mean?

I am looking at some lecture slides on a data science course which can be found here: https://github.com/cs109/2015/blob/master/Lectures/01-Introduction.pdf I, unfortunately, cannot see the video ...
33
votes
6answers
5k views

Data mining: How should I go about finding the functional form?

I'm curious about repeatable procedures that can be used to discover the functional form of the function y = f(A, B, C) + error_term where my only input is a set of ...
33
votes
3answers
38k views

What are the measure for accuracy of multilabel data?

Consider a scenario where you are provided with KnownLabel Matrix and PredictedLabel matrix. I would like to measure the goodness of the PredictedLabel matrix against the KnownLabel Matrix. But the ...
33
votes
8answers
42k views

What math subjects would you suggest to prepare for data mining and machine learning?

I'm trying to put together a self-directed math curriculum to prepare for learning data mining and machine learning. This is motivated by starting Andrew Ng's machine learning class on Coursera and ...
32
votes
1answer
855 views

Are there statistical lessons from the “Bible Code” episode

Although this question is somewhat subjective, I hope it qualifies as a good subjective question according to the faq guidelines. It is based on a question that Olle Häggström asked me a year ago and ...
31
votes
1answer
26k views

Difference between standard and spherical k-means algorithms

I would like to understand, what is the major implementation difference between standard and spherical k-means clustering algorithms. In each step, k-means computes distances between element vectors ...
29
votes
2answers
6k views

Why are p-values misleading after performing a stepwise selection?

Let's consider for example a linear regression model. I heard that, in data mining, after performing a stepwise selection based on the AIC criterion, it is misleading to look at the p-values to test ...
29
votes
5answers
27k views

Are decision trees almost always binary trees?

Nearly every decision tree example I've come across happens to be a binary tree. Is this pretty much universal? Do most of the standard algorithms (C4.5, CART, etc.) only support binary trees? From ...
28
votes
3answers
18k views

LSA vs. PCA (document clustering)

I'm investigation various techniques used in document clustering and I would like to clear some doubts concerning PCA (principal component analysis) and LSA (latent semantic analysis). First thing - ...
27
votes
9answers
16k views

Statistics and data mining software tools for dealing with large datasets

Currently I have to analyze approximately 20M records and build prediction models. So far I have tried out Statistica, SPSS, RapidMiner and R. Among these Statistica seems to be most suitable to deal ...
26
votes
3answers
29k views

Negative binomial distribution vs binomial distribution

What is the difference between the negative binomial distribution and the binomial distribution? I tried reading online, and I found that the negative binomial distribution is used when data points ...
26
votes
2answers
19k views

What is the difference between a loss function and decision function?

I see that both functions are part of data mining methods such as Gradient Boosting Regressors. I see that those are separate objects too. How is the relationship between both in general?
26
votes
7answers
15k views

What is the daily job routine of the machine learning scientist?

I'm a master CS student in a German university now writing my thesis. I will be done in two months I have to make the very hard decision if I should continue with a PhD or find a job in the industry. ...
26
votes
1answer
16k views

Distant supervision: supervised, semi-supervised, or both?

"Distant supervision" is a learning scheme in which a classifier is learned given a weakly labeled training set (training data is labeled automatically based on heuristics / rules). I think that both ...
25
votes
2answers
22k views

If k-means clustering is a form of Gaussian mixture modeling, can it be used when the data are not normal?

I'm reading Bishop on EM algorithm for GMM and the relationship between GMM and k-means. In this book it says that k-means is a hard assign version of GMM. I'm wondering does that imply that if the ...
25
votes
2answers
23k views

Boosting: why is the learning rate called a regularization parameter?

The learning rate parameter ($\nu \in [0,1]$) in Gradient Boosting shrinks the contribution of each new base model -typically a shallow tree- that is added in the series. It was shown to dramatically ...
24
votes
8answers
24k views

Perform K-means (or its close kin) clustering with only a distance matrix, not points-by-features data

I want to perform K-means clustering on objects I have, but the objects aren't described as points in space, i.e. by objects x features dataset. However, I am able ...
24
votes
6answers
3k views

What are the disadvantages of using mean for missing values?

I have an assignment (Data Mining course) and there is a part which asks: "What are the disadvantages of using mean for missing values?" in Missing Value section. ...
24
votes
3answers
21k views

How to know whether the data is linearly separable?

The data has many features (e.g. 100) and the number of instances is like 100,000. The data is sparse. I want to fit the data using logistic regression or svm. How do I know whether features are ...
24
votes
3answers
33k views

What is one class SVM and how does it work?

I was using one class SVM, implemented in scikit-learn, for my research work. But I have no good understanding of this. Can anyone please give a simple, good explanation of one class SVM?
22
votes
2answers
7k views

Cross Validation (error generalization) after model selection

Note: Case is n>>p I am reading Elements of Statistical Learning and there are various mentions about the "right" way to do cross validation( e.g. page 60, page 245). Specifically, my question is how ...
22
votes
3answers
1k views

First step for big data ($N = 10^{10}$, $p = 2000$)

Suppose you are analyzing a huge data set at the tune of billions of observations per day, where each observation has a couple thousand sparse and possibly redundant numerical and categorial variables....
21
votes
6answers
16k views

What is the difference between data mining and statistical analysis?

What is the difference between data mining and statistical analysis? For some background, my statistical education has been, I think, rather traditional. A specific question is posited, research is ...
21
votes
5answers
3k views

New revolutionary way of data mining?

The following excerpt is from Schwager's Hedge Fund Market Wizzards (May 2012), an interview with the consistently successful hedge fund manager Jaffray Woodriff: To the question: "What are some of ...
21
votes
3answers
28k views

What is the practical difference between association rules and decision trees in data mining?

Is there a really simple description of the practical differences between these two techniques? Both seem to be used for supervised learning (though association rules can also handle unsupervised). ...
21
votes
2answers
382 views

“Interestingness” function for StackExchange questions

I am trying to put together a data-mining package for StackExchange sites and in particular, I am stuck in trying to determine the "most interesting" questions. I would like to use the question score, ...
20
votes
3answers
7k views

Difference between Factorization machines and Matrix Factorization?

I came across the term Factorization Machines in recommender systems. I know what Matrix Factorization is for recommender systems but never heard of Factorization Machines. So what's the difference?
20
votes
2answers
2k views

Where and why does deep learning shine?

With all the media talk and hype about deep learning these days, I read some elementary stuff about it. I just found that it is just another machine learning method to learn patterns from data. But my ...
19
votes
6answers
11k views

Programmer looking to break into machine learning field

I am a software developer (mostly .NET and Python about 5 years experience). What can I do to help me get a job in the machine learning field or really anything that will get me started in that field? ...
19
votes
2answers
5k views

How to predict when the next event occurs, based on times of previous events?

I'm a high school student and I'm working on a computer programming project, but I don't have a lot of experience in statistics and modeling data beyond a high school statistics course so I'm kinda ...
18
votes
3answers
43k views

What is the difference between bagging and random forest if only one explanatory variable is used?

" The fundamental difference between bagging and random forest is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset ...
18
votes
7answers
7k views

Biased Data in Machine Learning

I am working on a Machine Learning project with data that is already (heavily) biased by data selection. Let's assume you have a set of hard coded rules. How do you build a machine learning model to ...

1
2 3 4 5
24