Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...
0
votes
0answers
40 views
Feature Extraction
I have some tweets labelled as religion and education. I want to extract features from them so that I can make a model to predict future tweets. Can someone please suggest something??
0
votes
0answers
124 views
How do I know whether I need to normalize before ranking?
Update
Experimenting with descriptive statistics, I determined that if rank the scaled weighted average quantity for each set, computed as follows:
$$Mean(Weighted Quantity) * \frac {Median ( ...
0
votes
1answer
29 views
Doubts regarding Feature Selection for disease prediction
I am doing disease classification( ie a person is classified as normal or abnormal) using naivebayes and SMO classifier.I have around 30 attributes.Out of these I need to select the most relevant ...
0
votes
2answers
179 views
Time series prediction - what is Autoregressive Tree model ? (Python)
Our problem: model evolution of values of a continuous variable over time.
I came through a paper presenting an approach for predicting the next values for a time series. Whereas ARIMA model is more ...
1
vote
6answers
530 views
How to detect outliers in skewed data set?
I am working on my school datamining project. Within preprocessing stage I need to remove outliers from my data set which is positively skewed (see description). I have an idea to remove all values ...
0
votes
1answer
79 views
Hidden Markov Model - Confusion
I don't know whether this is the correct forum for this but here goes:
I'm trying to implement a Hidden Markov Model to be able to predict and find the best sequence/path for a training file.
So ...
1
vote
0answers
55 views
recommender system implicit rating to ordinal scale
There are 4 ways a user can show preference for an article within my news app: number of times an article was viewed; for how long was the article viewed; whether it was favorited or not; and number ...
0
votes
1answer
98 views
How to create artificial data with one binary response variable?
I want to check various classification model like random forest, tree, knn,etc. I used some bench marking data set but now I need to simulate my own data set with a binary response variable.
3
votes
1answer
216 views
Imputation with Random Forests
I have two questions on using random forest (specifically randomForest in R) for missing value imputation (in the predictor space).
1) How does the imputation algorithm work - specifically how and ...
-1
votes
1answer
67 views
Easy way to test usefulness of data for stock market analysis? [closed]
I have a lot of data (gigs) that may be useful in predicting equity prices. I can import these as a series of features (columns) in a table where the companies are rows. I have time series information ...
0
votes
0answers
51 views
the effects of feature matrix format on the training time of LIBSVM
I am using Libsvm to perform text classification tasks. I normally uses binary occurrence, TF/IDF to build feature set for the input documents. It normally takes quite longer for Libsvm to finish ...
1
vote
0answers
61 views
Determining optimal height for regression tree
I have a data set of approximately 400,000 records (for those of you who know, the data set is the one provided by yahoo for their yahoo learning to rank challenge). From this data set I learn a ...
2
votes
1answer
209 views
Comparing distributions as opposed to medians?
Assume a hypothetical scenario of two events. During event 1, I observe a set of values $[X_1, X_2,X_3,...,X_n]$. A physical phenomena occurs and this triggers event 2 for a short period of time and I ...
0
votes
0answers
42 views
When to use the raw dataset as opposed to a transformed dataset for computing divergence?
Let us assume I have a set of observations:
Dataset 1: Raw
$A = [a_1,a_2,a_3,a_4,...]$
$B = [b_1,b_2,b_3,b_4,...]$
One assumptions before I proceed: Range of the values that the random variable ...
5
votes
1answer
118 views
Binary classification of DNA motif sequences (bioinformatics)
I've been working on on a method for binary classification of DNA
sequences. In more detail, here is what the method does.
Given a family of DNA sequences, for example DNA sequence
motifs, I try to ...
10
votes
2answers
420 views
How far will self study get me?
I have never taken part in an official or structured data analysis or machine learning course (other than recent online offerings) and have learned most of what I know from reading and trying things ...
1
vote
0answers
30 views
Do we need to apply the same transformation of predictors on a test dataset?
This maybe a stupid question. But I hope someone could help me. I first divide the dataset into training (75%) and test (25%). Then fit a logistic regression model on training data set. When fitting ...
7
votes
1answer
251 views
Data mining techniques in Obama's campaign
I came across this article about the data mining team in Obama's reelection campaign. Unfortunately, the article is very fuzzy about the actual machinery of the statistical algorithms. However, it ...
2
votes
1answer
242 views
Missing values in GLM
I am applying glm on a data in which most of the values are NAs or blank. For example, in the example data produced below (4 predictors and one response variable), the default glm command will remove ...
0
votes
1answer
61 views
Are probability graphic models useful for predictive modelling?
Are Probability Graphic Models (say specifically Bayesian Networks) useful for predictive modelling in terms of large data (100,000 - 1,000,000 rows) and many variables (hundreds)?
Meaning, is this ...
2
votes
2answers
107 views
Increasing the sample size does not help the classification performance
I am training a SVM classifier based on a given document collections. I started from using 500 documents for training, then I add another 500 for training, and so on. In other words, I have three ...
4
votes
1answer
62 views
Get a representative vector from a large set, and compare it with samples
We're a small team of programmers and we're trying to solve a little problem, but we think we need some advices from professional mathematicians.
We want to know if a picture of a card is an Identity ...
3
votes
0answers
357 views
Weka J48 decision tree problem
I have a CSV dataset which contains mean (Numeric), spread (Numeric), review (string), ...
4
votes
0answers
47 views
(Nominal) raters with no gold standard
A friend of mine took a document and broke it up into parts, then asked 5 subject matter experts to classify each part into nominal category A, B, C, D, or E. (I'm not sure yet, but D may be "All of ...
1
vote
0answers
138 views
Validation error less than training error — implications?
I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
0
votes
1answer
162 views
Distant supervision: supervised, semi-supervised, or both?
"Distant supervision" is a learning scheme in which a classifier is learned given a weakly labeled training set (training data is labeled automatically based on heuristics / rules). I think that both ...
1
vote
0answers
26 views
Confusion related to a paper related to multivariate spatio-temporal data with missing attributes
I was reading this paper related to handling missing attributes of multivariate spatio temporal data. http://astro.temple.edu/~tua86150/Lou_IJCAI_11.pdf.
Here they have tried to exploit the spatial ...
0
votes
0answers
61 views
training data SEQUENCE vs. training data SET?
My question is related to calssification in DataMining. But I believe anyone who has good background in Math/statistics can answer it.
As you remember a discrete-value training data of one ...
3
votes
1answer
135 views
Machine learning predicted value
When we fit a generalized linear regression (e.g., logistic regression, gamma regression) we are estimating the population average Y given the predictors $X$ ( i.e., $E(Y | X)$ ).
When we fit a ...
2
votes
0answers
59 views
Using doctor's data to identify hospitalisations
I have access to two large medical datasets of observational records in the UK. The first - Clinical Practice Research Datalink (CPRD) - has data on 100,000's of patients - largely dates of doctor's ...
2
votes
1answer
123 views
Variational inference for nested Chinese restaurant process
I recently read paper by Chong Wang and David M. Blei "Variational Inference for the Nested Chinese Restaurant Process". And I couldn't understand the next part (from p.5):
The variational update ...
2
votes
0answers
63 views
Data Mining / ML applications in hydrodynamics?
I have a question about Scientific Data Mining.
Do you know successful case studies of applying Data Mining / Machine Learning techniques in hydrodynamics?
In general, does it make actually sense to ...
1
vote
2answers
167 views
Data analysis on multiuser environment
I have following environment that I'd like to analyze.
My company is divided in two teams:
Insert Team (composed by 150 persons)
Quality Check Team (composed by 10 persons)
Suppose that all ...
1
vote
0answers
21 views
Relative to a Population: Statistically Valid Method for Categorizing Something as “Other”?
I am working similarity searching of an HTTP Request relative to the last N number of days of requests my system has collected using locality sensitive hashing based on Moses Charikar's "Similarity ...
1
vote
2answers
102 views
Predictive Modeling question on Weka
I would like to predict the number of flu cases in the future using predictive modeling. I am very new to statistics, so I'm not sure which classifier to use in this case.
For the attributes, I'm ...
2
votes
0answers
37 views
Spectral Clustering of Graph
I am trying to cluster the graph using spectral clustering. However I am unaware of the number of classes that exist in the data.
Will it be a good idea to do PCA on the adjacency matrix to find ...
4
votes
0answers
79 views
Regarding the sampling procedure in Adaboost algorithm
The AdaBoost algorithm states that it is to train a classifier based on the training data according to a weight vector.
Assume the size of training data is N, the weight vector is of dimension N as ...
0
votes
0answers
67 views
curve fitting with extremely “large” numbers
here is an unconventional but apparently workable idea. wondering if anyone has tried something like it, esp looking for references, examples, or nearby related work.
am working on a mathematical ...
3
votes
1answer
84 views
Regarding redundant training data in building SVM-based classifier
To build a SVM-based classifier, I have a training data set consisting of N data points. Some of them are redundant. For instance, there have 50 data points which are exactly the same, and there have ...
3
votes
1answer
295 views
What is the practical difference between association rules and decision trees in data mining?
Is there a really simple description of the practical differences between these two techniques?
Both seem to be used for supervised learning (though association
rules can also handle unsupervised).
...
2
votes
1answer
105 views
LDA topic models for various and unknown domains
We have a large collection (1-2M) of documents of various domains (politics, design, programming, etc.). And lets assume that we don't know the exact number of domains. And our goal is to build ...
2
votes
2answers
276 views
Clustering high-dimensional sparse binary data
I am trying to cluster Facebook users based on their likes.
I have two problems: First, since there is no dislike in Facebook all I have is having likes (1) for some items but for the rest of the ...
3
votes
1answer
45 views
Methods for teasing apart the influence of different time series features on a target feature?
Are there any established methods for teasing apart the influence of different time series features on a target feature?
To illustrate:
The target: Sales volume of kittens.
Features: Time of year, ...
2
votes
1answer
114 views
Modelling longitudinal data
We have longitudinal data on children(n<20) in which we measure different quantities A,B,C,D (like distance walked, time spent in school etc.). These are all continuous variables. We measure these ...
3
votes
2answers
249 views
Mathematics behind classification and regression trees
Can anyone help explain some of the mathematics behind classification in CART? I'm looking to understand how two main stages happen. For instance I trained a CART classifier on a dataset and used a ...
0
votes
1answer
212 views
How to find outliers in a data series?
I have a series of 100 points
My dataset can be found here . Each row is a data series. The plot for 90th row is
It's easy to detect outliers visually by plotting example. I tried using hampel ...
2
votes
3answers
280 views
Which type of regression fits better?
I am a newbie in data mining world. I have a general question.
I have a data set which has 10 independent variables and one target variable named as category which has 9 values like: 1, 2, 3, 4, 5, 6, ...
2
votes
2answers
77 views
Anomaly Detection in a set of points
I have a set of points in a matrix of size 100 x 100(total 10000 points). I know that there are roughly 500 anomaly points in it. There is a corresponding truth file which contains the true anomalous ...
0
votes
3answers
78 views
How to use LOF for outlier detection as I have training and test dataset?
I want to use the Local Outlier Factor (LOF) algorithm for outlier detection but it simply finds outliers on unlabed data as whole and you do not need to have a training and test set. However in my ...
2
votes
0answers
30 views
What is the difference between emerging and discriminative patterns?
I'm working on a way to make efficient classification on sequential patterns mining algorithms. Looking in the state-of-the-art I found that there is two type of algorithm to address this problem : ...
