# Tagged Questions

'Big data' refers to situations where the number of observations is so large that it necessitates changes in the way the the data analyst thinks about or conducts the analysis.

19 views

### Large scale k nearest neighbor search

For example we have n samples with vector length k (n>>k). And we can't load this matrix in RAM at once. Is there any solutions for large scale nearest neighbor search? any libs suitable for this? ...
46 views

### Algorithms for regression analysis which can handle large scale datasets

I am a CS undergraduate student and for my final project i developed a regression algorithm that is suited for large-scale datasets (i wouldn't say 'Big Data', but still large scale). For the final ...
26 views

### Question about the relevance of random effects in the big panel-data context

In general, or at least as I know, the "estimation" of random effects does control for the individual/group/... specific variation. Hence by controlling for this variation by a random intercept- or ...
61 views

### How big of a dataset can R or SAS handle for regression?

On a standard computer (~3-6gb RAM) with 2 - 4 processors. What are the size restrictions on a matrix of data for R or SAS?
182 views

### Can a sample be too large for ANOVA or a t-test?

I have close to a million data sets and whenever I run mean comparison test, either ANOVA or a t-test, I get a significance level of less than .0001 on SPSS. I'm concerned that my sample is so large ...
89 views

### Dimensionality Reduction Algorithm for Large Dataset?

I have a reasonably large (5k variables x 120k cases) that I'd like to run a dimensionality reduction algorithm on. I tried doing a simple Factor Analysis on it in SPSS, but it (predictably) barfed on ...
148 views

### Cluster Big Data in R and Is Sampling Relevant?

I'm new to data science and have a problem finding clusters in a data set with 200,000 rows and 50 columns in R. Since the data have both numeric and nominal variables, methods like K-means which ...
254 views

### R as an alternative to SAS for large data

I know that R is not particularly helpful for analysing large datasets given that R loads all the data in memory whereas something like SAS does sequential analysis. That said, there are packages like ...
84 views

### I have GBs of Event-Based Data. How do I figure out causation?

I have a lot of event-based data about users of our website. For example, data in the format (verb, timestamp). There's about 10 or so different verbs (call them A, B, C, etc). I'm interested in ...
83 views

### Sampled median's accuracy

I'm working on a problem where I need to calculate the median for a very large data set (for instance, 100M values) that has a log-normal distribution. Because of the data set's size, we were ...
209 views

### Big Data vs multiple hypothesis testing?

Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics ...
21 views

### Massive Textual Corpus [duplicate]

Possible Duplicate: Where to find a large text corpus? I know someone has asked a similar question here, but I'm wondering whether anyone knows of a large textual corpus that is available ...
59 views

### Using doctor's data to identify hospitalisations

I have access to two large medical datasets of observational records in the UK. The first - Clinical Practice Research Datalink (CPRD) - has data on 100,000's of patients - largely dates of doctor's ...
204 views

### Algorithm for price optimization [closed]

I'm trying to figure out a way for calculating price optimization in a commerce environment. In other words, I'm trying to analyze how a company can increase revenue and profitability by analyzing ...
481 views

### Regression analysis for a massive dataset

I have a massive dataset, including about 5,000,000 points. There are 4 independent variables and two highly correlated dependent variables. How should I do the regression analysis? ...
57 views

### What are the steps for classification task with such a huge data?

I am working on the dataset KDD'99. It has 4 main classes, 39 features and about 3 million instances with very unbalance distribution over classes. As a newbie, I am curious about what are correct ...
128 views

### Clusters produced by R intersect

I am new here - and relatively new to statistics, data mining and R. I am trying to understand why my data is not clustering correctly - or if I am reading it wrong. Shortly about the project: My ...
163 views

### Best platform for running (python and ( R or Octave)) algorithms for (large/big) data analytics [closed]

I have a machine learning algorithm currently implemented in R, wrapped in python (rpy2). I would like to deploy this inside a web application and I am looking for the right platform to do this, ...
2k views

### Is sampling relevant in the time of 'big data'?

Or more so "will it be"? Big Data makes statistics and relevant knowledge all the more important but seems to underplay Sampling Theory. I've seen this hype around 'Big Data' and can't help wonder ...
359 views

### Significance test for large sample sizes

This is more of a theoretical question. Super large sample sizes will almost always show a significance when a $\chi^2$ test is done. Is there any other statistical test of significance (an ...
463 views

### k-fold cross-validation strategy for large data set in statistical learning

I'm trying to learn the Bayesian network structure from a very large data set, and the R package I used for learning can only handle a very small portion of the data set (~10%) at one time due to the ...
115 views

### What are Effective Regression Techniques for Linguistic Analysis of Linked Data?

Cross-post from MathOverflow where it was suggested that I might get better results here. I am in the early stages of a problem that involves parsing a large number ($\approx 5 \times 10^9$) of ...