Tagged Questions

0
votes
0answers
53 views

AIC vs BIC vs MDL

I am trying to learn the difference between the three approaches and their applications. a) As I understand, AIC = -LL+K BIC = -LL+(K*logN)/2 Unless I am ...
0
votes
2answers
79 views

How to interpret the output of the summary method for an lm object in R? [duplicate]

I am using sample algae data to understand data mining a bit more. I have used the following commands: ...
2
votes
0answers
37 views

SVM classifier (with soft-margin) implementation in R, gamma value and quadprog

I'm trying to implement a Support Vector Machine classifier in R and I have to solve the optimization problem using the quadprog R package which solves problems of the form : $$min_b \frac{1}{2} ...
2
votes
1answer
69 views

What does it imply when an estimate is not inside its 95% confidence interval?

What does it actually imply when a 95% CI does not contain an estimate (coefficient or parameter). Is there some model assumption that has not been satisfied? Or it means something else? I know when ...
0
votes
0answers
20 views

How does R{MASS} lda function use MLEs to improve its result?

I am using the LDA function in the MASS package of R, which has the following specification: ...
0
votes
0answers
49 views

Variables selection (continuous and classification): how to do in R?

My dataset have both classification (categorical) and continuous variables, ~ 300 variables in all. I'm looking for a way to reduce my attributes to be less than 300 and put them in the decision tree ...
0
votes
0answers
32 views

Error doing MARS with R and the polspline package

I hope I'm asking this in the right place, I was unsure wether to write here or on StackOverflow, but I suspect my problem has more to do with my ignorance in statistics than anything else, so I'll ...
0
votes
0answers
43 views

When to use the raw dataset as opposed to a transformed dataset for computing divergence?

Let us assume I have a set of observations: Dataset 1: Raw $A = [a_1,a_2,a_3,a_4,...]$ $B = [b_1,b_2,b_3,b_4,...]$ One assumptions before I proceed: Range of the values that the random variable ...
2
votes
1answer
251 views

Missing values in GLM

I am applying glm on a data in which most of the values are NAs or blank. For example, in the example data produced below (4 predictors and one response variable), the default glm command will remove ...
4
votes
0answers
80 views

Regarding the sampling procedure in Adaboost algorithm

The AdaBoost algorithm states that it is to train a classifier based on the training data according to a weight vector. Assume the size of training data is N, the weight vector is of dimension N as ...
2
votes
2answers
282 views

Clustering high-dimensional sparse binary data

I am trying to cluster Facebook users based on their likes. I have two problems: First, since there is no dislike in Facebook all I have is having likes (1) for some items but for the rest of the ...
2
votes
3answers
287 views

Which type of regression fits better?

I am a newbie in data mining world. I have a general question. I have a data set which has 10 independent variables and one target variable named as category which has 9 values like: 1, 2, 3, 4, 5, 6, ...
1
vote
0answers
63 views

Growing a classification tree manually

I want to create some CARTs in R. I used Clementine a while ago for this purpose and remember I could go into "manual mode" and grow the trees by hand. I could manually choose which variable to use ...
2
votes
1answer
110 views

R packages or open source software for training Hidden Markov chains

Are there any well-designed R packages or other open-source software for training Hidden Markov chains?
1
vote
2answers
195 views

Temporal association rules with arulesSequences with quantities

I am trying to mine product-usage sequences for multiple users of online gaming site. I have found the R package arulesSequences but am not sure how to fit it to my problem. The data format would be ...
6
votes
3answers
202 views

How to add outliers to an existing data?

I want to test few similarity measures for outlier detection. I've got some data from UCI repository, for example: Breast-Cancer. Is there a smart way to add artificial outliers to an existing data? ...
9
votes
2answers
192 views

Determining largest contributor in a group

I dont know much about statistics so bear with me. Lets say I have a set of 1000 workers. I want to figure out who the hardest worker is but I can only measure the amount of work getting done in ...
2
votes
1answer
462 views

Plotting a heatmap based on clustering in R

...
1
vote
0answers
39 views

Sequence mining in univariate discrete data using R [duplicate]

Possible Duplicate: r sequence recognition for univariate data sequence mining arules I am looking for a method to detect sequences within univariate discrete data without specifying the ...
2
votes
4answers
149 views

r sequence recognition for univariate data sequence mining

I am looking for a method to detect sequences within univariate discrete data without specifying the length of the sequence or the exact nature of the sequence beforehand (see e.g. Wikipedia - ...
3
votes
3answers
154 views

How to graphically reveal data structures?

I have a dataframe with 8 columns (variables, X1 to X8) . Each column represents a different model parameter. Each row of the dataframe represents a modelling scenario. All the modelling scenarios ...
2
votes
1answer
164 views

Are randomForest variable importance values comparable across same variables on different dates?

Are randomForest variable importance comparable across same variables on different dates? I have a data array X which is of size $T\times N\times K$, where $T=1500$, $N=1500$ and $K=10$. ...
2
votes
2answers
237 views

Variable selection in large datasets

I'm looking for an overview of some methods of variable selection. I use datasets with around 6000 variables (the level of missing values is satisfying i.e. there are no variables with 100% missing ...
2
votes
1answer
933 views

Logistic regression on categorical data

I have large dataset (around 2 million records and 300 features) with a lot of missing data. Most of the independent variables are categorical (some of these variables have more than 40 valid values). ...
11
votes
0answers
317 views

First step for big data ($N = 10^{10}$, $p = 2000$)

Suppose you are analyzing a huge data set at the tune of billions of observations per day, where each observation has a couple thousand sparse and possibly redundant numerical and categorial ...
1
vote
1answer
117 views

Regarding the R packages that share the similar functionalities of NLTK toolkit

Are there any R packages that share the similar functionalities of NLTK toolkit?
5
votes
3answers
845 views

Outliers spotting in time series analysis, should I pre-process data or not?

My question builds on a previous post on outlier detection in generic time series, and specifically on the answer provided by the always great Rob H. I work for a small-sized manufacturing company ...
8
votes
5answers
446 views

Data mining papers/examples

I am looking for some medium to long length papers/websites/etc about data mining, specifically where one dataset is explored in depth from data preparation through final model. I am particularly ...
3
votes
1answer
963 views

Finding suitable rules for new data using arules

I am using R (and the arules package) to mining transactions for association rules. What I wish to do is construct the rules and then apply them to new data. For example, say I have many rules, one ...
2
votes
1answer
151 views

Time series digitization

I am looking for a viable method to create an alphabet out of multidimensional time series data. My intention is to use pattern/string matching algorithms (potentially genetic) to find profitable ...
3
votes
1answer
402 views

Overcoming memory constraints in rpart?

I have 4.4 million observations, 160 binary features, and a binary response. Using rpart on Windows (64-bit, with the 64-bit R v2.13.0 build), I run out of memory on a machine with 64GB RAM. My memory ...
2
votes
0answers
529 views

How do I interpret this Weibull plot?

I was exploring Weibull analysis for understanding reliability of two specific specimen. I used the R package, "weibulltoolkit" and "survival" to get the plot in question. The dataset is big so I am ...
2
votes
1answer
599 views

What does this blur around the line mean in this graph?

I was playing around with ggplot2 using the following commands to fit a line to my data: ...
0
votes
0answers
158 views

Systematic way of choosing a threshold?

I am performing some analysis on understanding various characteristics of a certain population. To narrow down my selection (as I will be going through them manually), I am using different thresholds ...
5
votes
1answer
602 views

Generate random strings based on regular expressions in R

I'm trying to generate random textual data based on regular expressions. I'd like to be able to do this in R, as I know that R does have regex capabilities. Any leads? This question has come up ...
9
votes
1answer
397 views

Documented/reproducible examples of successful real-world applications of econometric methods?

This question might sound very broad, but here is what I am looking for. I know there are many excellent books about econometric methods, and many excellent expository articles about econometric ...
6
votes
6answers
1k views

Algorithms and methods for attribute/feature selection?

I have data with continuous class and I'm searching for good methods to reduce number of attributes. Now I'm using correlation based filters, random forests and Gram–Schmidt algorithm. What I want ...