Questions tagged [data-mining]
Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.
1,183
questions
440
votes
5
answers
171k
views
How to understand the drawbacks of K-means
K-means is a widely used method in cluster analysis. In my understanding, this method does NOT require ANY assumptions, i.e., give me a dataset and a pre-specified number of clusters, k, and I just ...
- 6,709
222
votes
13
answers
202k
views
What is the difference between data mining, statistics, machine learning and AI?
What is the difference between data mining, statistics, machine learning and AI?
Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different ...
178
votes
4
answers
160k
views
Cohen's kappa in plain English
I am reading a data mining book and it mentioned the Kappa statistic as a means for evaluating the prediction performance of classifiers. However, I just can't understand this. I also checked ...
- 8,131
140
votes
9
answers
58k
views
Obtaining knowledge from a random forest
Random forests are considered to be black boxes, but recently I was thinking what knowledge can be obtained from a random forest?
The most obvious thing is the importance of the variables, in the ...
- 3,954
92
votes
7
answers
41k
views
Euclidean distance is usually not good for sparse data (and more general case)?
I have seen somewhere that classical distances (like Euclidean distance) become weakly discriminant when we have multidimensional and sparse data. Why? Do you have an example of two sparse data ...
- 2,579
82
votes
9
answers
14k
views
Skills hard to find in machine learners?
It seems that data mining and machine learning became so popular that now almost every CS student knows about classifiers, clustering, statistical NLP ... etc. So it seems that finding data miners is ...
78
votes
2
answers
120k
views
Performance metrics to evaluate unsupervised learning
With respect to the unsupervised learning (like clustering), are there any metrics to evaluate performance?
- 2,717
73
votes
11
answers
45k
views
Having a job in data-mining without a PhD
I've been very interested in data-mining and machine-learning for a while, partly because I majored in that area at school, but also because I am truly much more excited trying to solve problems that ...
66
votes
2
answers
36k
views
Why only three partitions? (training, validation, test)
When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset.
This is because the models usually have ...
- 7,596
64
votes
3
answers
64k
views
Clustering with K-Means and EM: how are they related?
I have studied algorithms for clustering data (unsupervised learning): EM, and k-means.
I keep reading the following :
k-means is a variant of EM, with the assumptions that clusters are
...
- 793
62
votes
12
answers
20k
views
Software needed to scrape data from graph [closed]
Anybody have any experience with software (preferably free, preferably open source) that will take an image of data plotted on cartesian coordinates (a standard, everyday plot) and extract the ...
55
votes
8
answers
13k
views
Is sampling relevant in the time of 'big data'?
Or more so "will it be"? Big Data makes statistics and relevant knowledge all the more important but seems to underplay Sampling Theory.
I've seen this hype around 'Big Data' and can't help wonder ...
- 14.2k
54
votes
3
answers
2k
views
Do we have a problem of "pity upvotes"?
I know, this may sound like it is off-topic, but hear me out.
At Stack Overflow and here we get votes on posts, this is all stored in a tabular form.
E.g.:
post id voter id vote type ...
- 619
47
votes
5
answers
76k
views
Lift measure in data mining
I searched many websites to know what exactly lift will do? The results that I found all were about using it in applications not itself.
I know about the support and confidence function. From ...
- 625
45
votes
3
answers
48k
views
What are the differences between hidden Markov models and neural networks?
I'm just getting my feet wet in statistics so I'm sorry if this question does not make sense. I have used Markov models to predict hidden states (unfair casinos, dice rolls, etc.) and neural networks ...
- 693
44
votes
2
answers
263k
views
How to interpret the output of the summary method for an lm object in R? [duplicate]
I am using sample algae data to understand data mining a bit more. I have used the following commands:
...
- 593
43
votes
1
answer
47k
views
Relative variable importance for Boosting
I'm looking for an explanation of how relative variable importance is computed in Gradient Boosted Trees that is not overly general/simplistic like:
The measures are based on the number of times a ...
- 6,052
43
votes
3
answers
50k
views
What are the measure for accuracy of multilabel data?
Consider a scenario where you are provided with KnownLabel Matrix and PredictedLabel matrix. I would like to measure the goodness of the PredictedLabel matrix against the KnownLabel Matrix.
But the ...
- 4,337
41
votes
2
answers
7k
views
How to draw valid conclusions from "big data"?
"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
41
votes
1
answer
1k
views
Are there statistical lessons from the "Bible Code" episode
Although this question is somewhat subjective, I hope it
qualifies
as a good subjective question according to the faq guidelines.
It is based on a question that Olle Häggström asked me a year ago
and ...
- 291
37
votes
5
answers
37k
views
Are decision trees almost always binary trees?
Nearly every decision tree example I've come across happens to be a binary tree. Is this pretty much universal? Do most of the standard algorithms (C4.5, CART, etc.) only support binary trees? From ...
- 4,661
35
votes
3
answers
40k
views
Negative binomial distribution vs binomial distribution
What is the difference between the negative binomial distribution and the binomial distribution?
I tried reading online, and I found that the negative binomial distribution is used when data points ...
- 616
35
votes
5
answers
5k
views
Think like a bayesian, check like a frequentist: What does that mean?
I am looking at some lecture slides on a data science course which can be found here:
https://github.com/cs109/2015/blob/master/Lectures/01-Introduction.pdf
I, unfortunately, cannot see the video ...
- 4,520
35
votes
6
answers
6k
views
Data mining: How should I go about finding the functional form?
I'm curious about repeatable procedures that can be used to discover the functional form of the function y = f(A, B, C) + error_term where my only input is a set of ...
- 409
33
votes
4
answers
62k
views
What is one class SVM and how does it work?
I was using one class SVM, implemented in scikit-learn, for my research work. But I have no good understanding of this.
Can anyone please give a simple, good explanation of one class SVM?
33
votes
1
answer
32k
views
Difference between standard and spherical k-means algorithms
I would like to understand, what is the major implementation difference between standard and spherical k-means clustering algorithms.
In each step, k-means computes distances between element vectors ...
- 1,279
33
votes
8
answers
43k
views
What math subjects would you suggest to prepare for data mining and machine learning?
I'm trying to put together a self-directed math curriculum to prepare for learning data mining and machine learning. This is motivated by starting Andrew Ng's machine learning class on Coursera and ...
30
votes
2
answers
9k
views
Why are p-values misleading after performing a stepwise selection?
Let's consider for example a linear regression model. I heard that, in data mining, after performing a stepwise selection based on the AIC criterion, it is misleading to look at the p-values to test ...
- 2,057
30
votes
3
answers
38k
views
How to know whether the data is linearly separable?
The data has many features (e.g. 100) and the number of instances is like 100,000. The data is sparse. I want to fit the data using logistic regression or svm. How do I know whether features are ...
- 523
30
votes
1
answer
20k
views
Distant supervision: supervised, semi-supervised, or both?
"Distant supervision" is a learning scheme in which a classifier is learned given a weakly labeled training set (training data is labeled automatically based on heuristics / rules). I think that both ...
- 1,327
29
votes
8
answers
32k
views
Perform K-means (or its close kin) clustering with only a distance matrix, not points-by-features data
I want to perform K-means clustering on objects I have, but the objects aren't described as points in space, i.e. by objects x features dataset. However, I am able ...
- 293
29
votes
2
answers
22k
views
What is the difference between a loss function and decision function?
I see that both functions are part of data mining methods such as Gradient Boosting Regressors. I see that those are separate objects too.
How is the relationship between both in general?
29
votes
3
answers
33k
views
Boosting: why is the learning rate called a regularization parameter?
The learning rate parameter ($\nu \in [0,1]$) in Gradient Boosting shrinks the contribution of each new base model -typically a shallow tree- that is added in the series. It was shown to dramatically ...
- 348
29
votes
3
answers
20k
views
LSA vs. PCA (document clustering)
I'm investigation various techniques used in document clustering and I would like to clear some doubts concerning PCA (principal component analysis) and LSA (latent semantic analysis).
First thing - ...
- 1,279
28
votes
9
answers
16k
views
Statistics and data mining software tools for dealing with large datasets
Currently I have to analyze approximately 20M records and build prediction models. So far I have tried out Statistica, SPSS, RapidMiner and R. Among these Statistica seems to be most suitable to deal ...
28
votes
7
answers
15k
views
What is the daily job routine of the machine learning scientist?
I'm a master CS student in a German university now writing my thesis. I will be done in two months I have to make the very hard decision if I should continue with a PhD or find a job in the industry.
...
26
votes
3
answers
75k
views
What is the difference between bagging and random forest if only one explanatory variable is used?
" The fundamental difference between bagging and random forest is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset ...
26
votes
2
answers
25k
views
If k-means clustering is a form of Gaussian mixture modeling, can it be used when the data are not normal?
I'm reading Bishop on EM algorithm for GMM and the relationship between GMM and k-means.
In this book it says that k-means is a hard assign version of GMM. I'm wondering does that imply that if the ...
- 527
25
votes
6
answers
6k
views
What are the disadvantages of using mean for missing values?
I have an assignment (Data Mining course) and there is a part which asks: "What are the disadvantages of using mean for missing values?" in Missing Value section.
...
- 251
25
votes
4
answers
11k
views
Difference between Factorization machines and Matrix Factorization?
I came across the term Factorization Machines in recommender systems. I know what Matrix Factorization is for recommender systems but never heard of Factorization Machines. So what's the difference?
- 8,131
25
votes
3
answers
32k
views
What is the practical difference between association rules and decision trees in data mining?
Is there a really simple description of the practical differences between these two techniques?
Both seem to be used for supervised learning (though association
rules can also handle unsupervised).
...
- 899
24
votes
2
answers
9k
views
Cross Validation (error generalization) after model selection
Note: Case is n>>p
I am reading Elements of Statistical Learning and there are various mentions about the "right" way to do cross validation( e.g. page 60, page 245). Specifically, my question is how ...
- 7,990
23
votes
5
answers
3k
views
New revolutionary way of data mining?
The following excerpt is from Schwager's Hedge Fund Market Wizzards (May 2012), an interview with the consistently successful hedge fund manager Jaffray Woodriff:
To the question: "What are some of ...
- 6,066
22
votes
3
answers
1k
views
First step for big data ($N = 10^{10}$, $p = 2000$)
Suppose you are analyzing a huge data set at the tune of billions of observations per day, where each observation has a couple thousand sparse and possibly redundant numerical and categorial variables....
- 1,985
22
votes
2
answers
391
views
"Interestingness" function for StackExchange questions
I am trying to put together a data-mining package for StackExchange sites and in particular, I am stuck in trying to determine the "most interesting" questions. I would like to use the question score, ...
- 403
21
votes
6
answers
17k
views
What is the difference between data mining and statistical analysis?
What is the difference between data mining and statistical analysis?
For some background, my statistical education has been, I think, rather traditional. A specific question is posited, research is ...
- 5,896
21
votes
5
answers
18k
views
Are Random Forest and Boosting parametric or non-parametric?
By reading the excellent Statistical modeling: The two cultures (Breiman 2001), we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning ...
- 6,052
21
votes
3
answers
32k
views
Do I need to drop variables that are correlated/collinear before running kmeans?
I am running kmeans to identify clusters of customers. I have approximately 100 variables to identify clusters. Each of these variables represent the % of spend by a customer on a category. So, if I ...
- 211
20
votes
2
answers
2k
views
Where and why does deep learning shine?
With all the media talk and hype about deep learning these days, I read some elementary stuff about it. I just found that it is just another machine learning method to learn patterns from data. But my ...
- 8,131
20
votes
1
answer
17k
views
Relationship between Hessian Matrix and Covariance Matrix
While I am studying Maximum Likelihood Estimation, to do inference in Maximum Likelihood Estimaion, we need to know the variance. To find out the variance, I need to know the Cramer's Rao Lower Bound, ...
- 1,623