Questions tagged [data-mining]
Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.
1,184
questions
2
votes
0
answers
197
views
Choosing between Latent Dirichlet Allocation and Association Rule Mining
Data Mining and Statistical Analysis has a general discussion on stats vs data mining. If I may, narrow down the question a bit - are there any general demarcations that allows you to decide which ...
- 451
9
votes
2
answers
5k
views
Computer game datasets
I have been looking for computer game datasets, but so far I've only been able to find the 'Avatar History' dataset for WoW.
Are there any other interesting datasets out there, possibly for other ...
1
vote
0
answers
299
views
The relationship between machine learning, data mining and statistical analysis? [duplicate]
Possible Duplicate:
What is the difference between data mining, statistics, machine learning and AI?
How to compare
machine learning vs. data mining?
data mining vs. statistical analysis?
...
- 2,687
8
votes
2
answers
3k
views
Generate random strings based on regular expressions in R
I'm trying to generate random textual data based on regular expressions. I'd like to be able to do this in R, as I know that R does have regex capabilities. Any leads?
This question has come up ...
- 193
10
votes
2
answers
2k
views
Getting started with biclustering
I have been doing some casual internet research on biclusters. (I have read the Wiki article several times.) So far, it seems as if there are few definitions or standard terminology.
I was ...
- 1,619
5
votes
1
answer
257
views
Software for ML/NLP/DM laboratory
suppose you might buy some software to be used in a ML/NLP/DM laboratory. What software package would you ask for? Let's say: MATLAB (with some toolboxes), SPSS, and what else?
I know that there is a ...
17
votes
3
answers
1k
views
When is interactive data visualization useful to use?
While preparing for a talk I will give soon, I recently started digging into two major (Free) tools for interactive data visualization: GGobi and mondrian - both offer a great range of capabilities (...
- 20.8k
11
votes
1
answer
1k
views
Documented/reproducible examples of successful real-world applications of econometric methods?
This question might sound very broad, but here is what I am looking for. I know there are many excellent books about econometric methods, and many excellent expository articles about econometric ...
- 326
13
votes
2
answers
501
views
Exploratory analysis of spatio-temporal forecast errors
The data: I have worked recently on analysing the stochastic properties of a spatio-temporal field of wind power production forecast errors. Formally, it can be said to be a process $$ \left (\epsilon^...
- 6,425
41
votes
1
answer
1k
views
Are there statistical lessons from the "Bible Code" episode
Although this question is somewhat subjective, I hope it
qualifies
as a good subjective question according to the faq guidelines.
It is based on a question that Olle Häggström asked me a year ago
and ...
- 291
4
votes
1
answer
956
views
Which method to use for anomaly detection?
I think people here could guide me in solving a problem related to anomaly detection in Computer Science. The term anomaly here refers to some undesired event occuring in the system like a virus ...
- 353
3
votes
3
answers
3k
views
Analyzing the difference between two datasets where one is a subset of the other
I apologize in advance for the vague title, but I couldn't think of anything better.
I have two datasets, where one is a very small subset of the other. The percentage of people who have a specific ...
Sam Thropton
24
votes
2
answers
9k
views
Cross Validation (error generalization) after model selection
Note: Case is n>>p
I am reading Elements of Statistical Learning and there are various mentions about the "right" way to do cross validation( e.g. page 60, page 245). Specifically, my question is how ...
- 7,990
12
votes
14
answers
2k
views
How much information can you mine out of a name?
A name: first, possibly a middle, and surname.
I'm curious about how much information you can mine out of a name, using publicly available datasets. I know that you can get the following with ...
6
votes
2
answers
360
views
Is there a bias correction for effect size in a data mining context?
Given $K$ possible 'treatments' of some kind, and independent observations of some response under those treatments, say $X_{i,k}$ for $i=1,\ldots,n_k$ and $k=1,\ldots,K$, I am faced with the classical ...
- 14.2k
9
votes
4
answers
20k
views
Reducing false positive rate
I am running a model for a problem in insurance domain. The final results show some false positive x and some false negative y. I am using SAS Enterprise Miner for this. Can somebody suggest me how to ...
- 1,597
3
votes
1
answer
199
views
Extrapolating the amount of data missing from the amount of data partially missing
Suppose I have a fleet of two-car trains riding around, and that each car is equipped with a data recording device. Unfortunately, some of the recording devices aren't working. I don't know either the ...
user2491
2
votes
3
answers
243
views
Methods of grouping sets of data
I have data which has several properties (metadata, as key value pairs, where the keyspace is shared over the whole dataset) per object.
I took a sample of objects and divided them in n groups ...
- 121
6
votes
2
answers
295
views
Discerning between two different linear regression models in one sample
Suppose I observe a sample $(y_i,x_i)$, $i=1,...,n$. Suppose that I know the following:
$y_i=\alpha_0+\alpha_1x_i+\varepsilon_i$, $i \in J\subset\{1,...,n\}$
$y_i=\beta_0+\beta_1x_i+\varepsilon_i$, ...
- 34.2k
222
votes
13
answers
202k
views
What is the difference between data mining, statistics, machine learning and AI?
What is the difference between data mining, statistics, machine learning and AI?
Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different ...
4
votes
2
answers
535
views
How to measure/weight the importance of tags?
Lets say we have $T$ tags and $N$ articles and lets say that for each tag $t_{i}$ we know that it has tagged $n_{i}$ articles. Meaning that, frequency($t_{i}$)=$n_{i}$.
Given the above information ...
user1975
6
votes
1
answer
786
views
Data mining algorithm suggestion
I would like to use data mining to try to find a good workout schemes. The input dataset will contain the parameters of a set of workouts with dates and different performance and medical measures. The ...
- 495
28
votes
9
answers
16k
views
Statistics and data mining software tools for dealing with large datasets
Currently I have to analyze approximately 20M records and build prediction models. So far I have tried out Statistica, SPSS, RapidMiner and R. Among these Statistica seems to be most suitable to deal ...
15
votes
3
answers
14k
views
Flowcharts to help selecting the proper analysis technique and test
As someone who needs statistical knowledge but is not a formally trained statistician, I'd find it helpful to have a flowchart (or some kind of decision tree) to help me choose the correct approach to ...
16
votes
10
answers
1k
views
A survey of data-mining software tools
Although I was trained as an engineer, I find that I'm becoming more interested in data mining. Right now I'm trying to investigate the field further. In particular, I would like to understand the ...
8
votes
10
answers
597
views
Data mining conferences? [closed]
What are the most significant annual Data Mining conferences?
Rules:
One conference per answer
Include a link to the conference
21
votes
6
answers
17k
views
What is the difference between data mining and statistical analysis?
What is the difference between data mining and statistical analysis?
For some background, my statistical education has been, I think, rather traditional. A specific question is posited, research is ...
- 5,896
6
votes
2
answers
443
views
How to limit my input data for Jaccard item-item similarity calculation?
I'm trying to compute item-item similarity using Jaccard (specifically Tanimoto) on a large list of data in the format
(userid, itemid)
An item is considered as ...
- 161
11
votes
2
answers
4k
views
What is the difference between functional data analysis and high dimensional data analysis
There are a lot of references in the statistic literature to "functional data" (i.e. data that are curves), and in parallel, to "high dimensional data" (i.e. when data are high dimensional vectors). ...
- 6,425
13
votes
4
answers
7k
views
Best ways to aggregate and analyze data
Having just recently started teaching myself Machine Learning and Data Analysis I'm finding myself hitting a brick wall on the need for creating and querying large sets of data. I would like to take ...
- 1,189
11
votes
6
answers
1k
views
Dubious use of signal processing principles to identify a trend
I am proposing to try and find a trend in some very noisy long term data. The data is basically weekly measurements of something which moved about 5mm over a period of about 8 months. The data is to ...
- 577
4
votes
2
answers
2k
views
What are the key statistical concepts that relate to data mining?
What topics in statistics are most useful/relevant to data mining?
9
votes
3
answers
1k
views
Data Mining-- how to tell whether the pattern extracted is meaningful?
I am sure that everyone who's trying to find patterns in historical stock market data or betting history would like to know about this. Given a huge sets of data, and thousands of random variables ...
- 985
11
votes
6
answers
3k
views
Algorithms and methods for attribute/feature selection?
I have data with continuous class and I'm searching for good methods to reduce number of attributes. Now I'm using correlation based filters, random forests and Gram–Schmidt algorithm.
What I want ...
- 463