Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...
10
votes
0answers
1k views
Machine learning self-learning book? [duplicate]
Possible Duplicate:
Machine learning cookbook / reference card / cheatsheet?
I wonder if there is a good self-learning textbook for machine learning? I am particularly looking for those in ...
2
votes
1answer
152 views
Time series digitization
I am looking for a viable method to create an alphabet out of multidimensional time series data. My intention is to use pattern/string matching algorithms (potentially genetic) to find profitable ...
5
votes
2answers
215 views
Sophisticated models for classifying short pieces of texts
I have about 30000 book names assigned to 6 categories, and I want to build scalable and accurate classifiers. So far I have only been able to use Naive Baye's and LibLINEAR classifiers and they both ...
4
votes
1answer
206 views
What are subjective interestingness measures?
I came across the word subjective interestingness measures in my book, where author says that:
Subjective interestingness measures are based on user belief in the
data. These measures find ...
1
vote
1answer
222 views
Dimension reduction issues in self-organizing maps (SOM)
Self organizing maps are claimed to be able to visualize/cluster high-dimensional data in a smaller dimensional space. I have some difficulties in understanding this statement.
Consider a ...
3
votes
1answer
441 views
Text feature vector extraction
I have a class assignment to implement a couple existing ways to extract feature vectors from a given set of texts, so they can be used to classify those texts using k-nearest neighbour algorithm. The ...
3
votes
0answers
253 views
What are some useful data visualisation or data mining techniques to investigate horse racing forms?
I have a dataset of 13k horse races from four different tracks with an average of 11 runners per race. In all, there are 26k unique runners in the data. For each race, I know who came first, second, ...
3
votes
1answer
421 views
Overcoming memory constraints in rpart?
I have 4.4 million observations, 160 binary features, and a binary response. Using rpart on Windows (64-bit, with the 64-bit R v2.13.0 build), I run out of memory on a machine with 64GB RAM. My memory ...
6
votes
5answers
431 views
What can I do beyond Pearson correlation?
While checking to see whether two variables were correlated, I observed that applying Pearson correlation yielded numbers as low as 0.1 indicating no correlation. Is there anything I can do to ...
2
votes
1answer
211 views
Figuring out probabilities with Hidden Markov Models
I'm really new to statistics so sorry in advance if this question does not make sense.
Background:
I'm trying to learn about hidden Markov models and they seem interesting but I was wondering about ...
3
votes
0answers
110 views
Data prep / variable creation for predictive models
I was reading a couple of the write ups from a Kaggle challenge: Here is one and another
and it got me wondering about variable creation in data mining and why there seems to exist so few texts or ...
9
votes
1answer
181 views
How can I find correlations between crashes and system environments?
In my free time, I am working on a small web based system which collects crash reports (but not other, non-crashing bug reports) that are sent from Delphi Windows applications.
For troubleshooting, ...
6
votes
3answers
2k views
Lift measure in data mining
I searched many websites to know what exactly lift will do? The results that I found all were about using it in applications not itself.
I know about the support and confidence function. From ...
7
votes
3answers
1k views
How to quickly select important variables from a very large dataset?
I have a dataset with about 2,000 binary variables/200,000 rows and I'm trying to predict a single binary dependent variable. My chief goal at this stage isn't getting accuracy of prediction, but ...
2
votes
2answers
720 views
Apriori algorithm in plain English?
I read wiki article about Apriori. I have the trouble in understanding the prune and Join step. Can anyone explain me how Apriori algorithm works in simple terms(such that Novice like me can ...
10
votes
1answer
359 views
How to predict when the next event occurs, based on times of previous events?
I'm a high school student and I'm working on a computer programming project, but I don't have a lot of experience in statistics and modeling data beyond a high school statistics course so I'm kinda ...
4
votes
2answers
301 views
Machine learning in web application?
There are a lot of outstanding machine learning/data mining standalone applications available in different languages like Java, Python, and others.
However, I wonder, practically, in case of applying ...
3
votes
1answer
142 views
Change and anomaly detection
I recently wrote a program that graphed data points so that a user could scroll through them and find "interesting" parts of the data.
Now I am looking at ways to make it even simpler by making a ...
-2
votes
1answer
438 views
Example to show that items in a strong association rule may actually be negatively correlated?
Give a short example to show that items in a strong association rule may actually be negatively correlated?
Thanks in advance!
4
votes
1answer
674 views
Best practices for measuring and avoiding overfitting?
I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting?
I ...
2
votes
0answers
62 views
Determine optimal (recurring) frequency of discrete values
So I'm trying to determine a semi-efficient way to calculate the optimal (recurring) frequency of a set of data. The data only exists at random time periods, but it is assumed that the data set is ...
0
votes
3answers
233 views
Looking to generate fun graphs/data for basketball matches
I'm looking to generate as much interesting data as I can from some 2on2 basketball games.
The only data I have recorded is the players on each team, and the final score.
What interesting things can ...
0
votes
3answers
68 views
Data sources related to Federal regulation
I'm looking for sources of data related to Federal regulation and enforcement. Things like the following:
Size of Federal regulatory agencies (number of workers)
Number of pages of Federal ...
3
votes
1answer
865 views
Bagging with oversampling for rare event predictive models
Does anyone know if the following has been described and (either way) if it sounds like a plausible method for learning a predictive model with a very unbalanced target variable?
Often in CRM ...
5
votes
1answer
189 views
Detect statistical anomalies
I've read these SO questions and this is not duplicate of them
best way to statistically detect anomalies in data
Statistical calculations
Algorithm(s) for spotting anomalies ("spikes") in ...
2
votes
0answers
541 views
How do I interpret this Weibull plot?
I was exploring Weibull analysis for understanding reliability of two specific specimen. I used the R package, "weibulltoolkit" and "survival" to get the plot in question. The dataset is big so I am ...
24
votes
5answers
1k views
Software needed to scrape data from graph
Anybody have any experience with software (preferably free, preferably open source) that will take an image of data plotted on cartesian coordinates (a standard, everyday plot) and extract the ...
0
votes
2answers
171 views
Ways of finding associations in time series
I am trying to develop a logic to identify association between different time series for association mining. I have a lot of series and need to find whether or not the association exists. I figured ...
0
votes
2answers
473 views
Examples of wrapping open source machine learning software in PMML?
Does anyone know of examples of wrapping an existing piece of supervised learning software to output models in PMML format? Of particular interest are learners that just take in labeled vectors of ...
2
votes
1answer
166 views
Is there a way to represent datasets using Predictive Model Markup Language?
This seems like a dumb question, but does PMML have a way to represent a data set? And if not, why not? There's detailed support for defining the name, type, and legal values of each feature that ...
1
vote
0answers
130 views
Entropy value for different sets of data
I am trying to compute the entropy for different sets of data and then compare the obtained values.
The idea is that the sets of data may change their size, I might have sets sizes ranging from 10 to ...
2
votes
1answer
611 views
What does this blur around the line mean in this graph?
I was playing around with ggplot2 using the following commands to fit a line to my data:
...
3
votes
2answers
4k views
Difference between experimental data and observational data?
I'm a novice to data mining and started to read about it. What's the exact difference between experimental data and observation data? Both are obviously data; and many say observation data can lead to ...
7
votes
5answers
1k views
Video lectures about data mining?
I want to learn data mining. Are there any free video lectures out there which explain the process of data mining in depth?
4
votes
2answers
417 views
Will the silhouette formula change depending on the distance metric?
I am using Silhouette width to compute the best value for k in k-means. As I am performing document clustering, I am calculating the values of a and ...
12
votes
4answers
549 views
How to begin reading about data mining?
I'm a novice who is going to start reading about data mining. I have basic knowledge of AI and statistics. Since many say that machine learning also plays an important role in data mining, is it ...
2
votes
1answer
82 views
calculate the rate of change
I posted this on maths, but seems it would be better here :S
http://math.stackexchange.com/questions/49941/calculate-the-rate-of-change
Basically
I am trying to calculate the change frequency for ...
0
votes
0answers
164 views
Systematic way of choosing a threshold?
I am performing some analysis on understanding various characteristics of a certain population. To narrow down my selection (as I will be going through them manually), I am using different thresholds ...
2
votes
1answer
89 views
What are the measure for accuracy of multilabel data?
Consider a scenario where you are provided with KnownLabel Matrix and PredictedLabel matrix. I would like to measure the goodness of the PredictedLabel matrix against the KnownLabel Matrix.
But the ...
0
votes
1answer
101 views
Notion of orthogonality in clustering
I'm reading in a paper "the goal is to cluster data in a way that is meaningful and at the same time as orthogonal as possible to the given classification"
Could someone have any idea how the notion ...
1
vote
2answers
91 views
Calculating the probability that demographic trends happened by chance
I ran an ad campaign on Facebook and I have impressions (ad-views), clicks, and sign-ups broken down by gender and age.
Certain demographics have a higher rate of clicks and sign-ups when compared to ...
2
votes
0answers
172 views
In what ways do data mining and machine learning competitions aid/detract from these academic fields and their commercial applications? [closed]
The Netflix competition kicked off a plethora of competitions to develop the best methods to learn information from given datasets. These include predicting traffic flow, predicting the success of ...
4
votes
3answers
635 views
Are decision trees almost always binary trees?
Nearly every decision tree example I've come across happens to be a binary tree. Is this pretty much universal? Do most of the standard algorithms (C4.5, CART, etc.) only support binary trees? From ...
1
vote
0answers
121 views
Measures of predictive power of attributes in data mining
What are the most widely used measures of predictive power of attributes in scoring models?
Motivation: I have a lot of attributes, more than I can study by myself and I want to select somehow the ...
2
votes
1answer
229 views
What is the association rule learning approach to the logical XOR problem?
The "exclusive or" function has a long and arduous history in the AI/machine learning communities. From my understanding of "association rule learning", xor would appear to be a problem for this type ...
3
votes
1answer
118 views
Use of survival analysis for trigger mining
I am tasked with analyzing data to find "triggers" to an event. Specifically, this is transaction data from a bank (e.g., checking account daily balances, daily over draft fees, daily number of checks ...
2
votes
1answer
1k views
Sliding window validation for time series
I have a broad question about sliding window validation. Specifically, I am looking at using Rapid Miner to predict future values of a financial series using "lagged" values of that series and other ...
0
votes
0answers
201 views
Machine learning questions (questions) [closed]
I have a fewquestions which i would appreciate if you can enlighten me in. Theese questions come from Past papers of Machine learning exams and i have no idea on how to solve any. you can set your ...
35
votes
4answers
1k views
Do we have a problem of “pity upvotes”?
I know, this may sound like it is off-topic, but hear me out.
At Stack Overflow and here we get votes on posts, this is all stored in a tabular form.
E.g.:
post id voter id vote type ...
0
votes
0answers
131 views
How to calculate variance of three or more attributes with known frequencies
Question was originally posted on Stack Overflow.
I want to calculate variance of data at a given time where I know the frequency of the conditional and decision attributes.
At time $t$ every ...
