'Big data' refers to situations where the number of observations is so large that it necessitates changes in the way the the data analyst thinks about or conducts the analysis.
0
votes
0answers
19 views
Large scale k nearest neighbor search
For example we have n samples with vector length k (n>>k).
And we can't load this matrix in RAM at once.
Is there any solutions for large scale nearest neighbor search? any libs suitable for this? ...
0
votes
1answer
46 views
Algorithms for regression analysis which can handle large scale datasets
I am a CS undergraduate student and for my final project i developed a regression algorithm that is suited for large-scale datasets (i wouldn't say 'Big Data', but still large scale).
For the final ...
0
votes
0answers
26 views
Question about the relevance of random effects in the big panel-data context
In general, or at least as I know, the "estimation" of random effects does control for the individual/group/... specific variation.
Hence by controlling for this variation by a random intercept- or ...
0
votes
0answers
61 views
How big of a dataset can R or SAS handle for regression?
On a standard computer (~3-6gb RAM) with 2 - 4 processors.
What are the size restrictions on a matrix of data for R or SAS?
5
votes
2answers
182 views
Can a sample be too large for ANOVA or a t-test?
I have close to a million data sets and whenever I run mean comparison test, either ANOVA or a t-test, I get a significance level of less than .0001 on SPSS. I'm concerned that my sample is so large ...
5
votes
1answer
89 views
Dimensionality Reduction Algorithm for Large Dataset?
I have a reasonably large (5k variables x 120k cases) that I'd like to run a dimensionality reduction algorithm on. I tried doing a simple Factor Analysis on it in SPSS, but it (predictably) barfed on ...
3
votes
2answers
148 views
Cluster Big Data in R and Is Sampling Relevant?
I'm new to data science and have a problem finding clusters in a data set with 200,000 rows and 50 columns in R.
Since the data have both numeric and nominal variables, methods like K-means which ...
7
votes
2answers
254 views
R as an alternative to SAS for large data
I know that R is not particularly helpful for analysing large datasets given that R loads all the data in memory whereas something like SAS does sequential analysis. That said, there are packages like ...
3
votes
1answer
84 views
I have GBs of Event-Based Data. How do I figure out causation?
I have a lot of event-based data about users of our website. For example, data in the format (verb, timestamp). There's about 10 or so different verbs (call them A, B, C, etc).
I'm interested in ...
4
votes
3answers
83 views
Sampled median's accuracy
I'm working on a problem where I need to calculate the median for a very large data set (for instance, 100M values) that has a log-normal distribution. Because of the data set's size, we were ...
4
votes
3answers
209 views
Big Data vs multiple hypothesis testing?
Nate Silver in his excellent "The Noise and the Signal" warned that we are much in awe of Big Data. But, that Big Data predictions in many fields have been disastrous (financial markets and economics ...
0
votes
0answers
21 views
Massive Textual Corpus [duplicate]
Possible Duplicate:
Where to find a large text corpus?
I know someone has asked a similar question here, but I'm wondering whether anyone knows of a large textual corpus that is available ...
2
votes
0answers
59 views
Using doctor's data to identify hospitalisations
I have access to two large medical datasets of observational records in the UK. The first - Clinical Practice Research Datalink (CPRD) - has data on 100,000's of patients - largely dates of doctor's ...
1
vote
1answer
204 views
Algorithm for price optimization [closed]
I'm trying to figure out a way for calculating price optimization in a commerce environment. In other words, I'm trying to analyze how a company can increase revenue and profitability by analyzing ...
3
votes
2answers
481 views
Regression analysis for a massive dataset
I have a massive dataset, including about 5,000,000 points. There are 4 independent variables and two highly correlated dependent variables.
How should I do the regression analysis?
...
0
votes
0answers
57 views
What are the steps for classification task with such a huge data?
I am working on the dataset KDD'99. It has 4 main classes, 39 features and about 3 million instances with very unbalance distribution over classes. As a newbie, I am curious about what are correct ...
2
votes
0answers
128 views
Clusters produced by R intersect
I am new here - and relatively new to statistics, data mining and R. I am trying to understand why my data is not clustering correctly - or if I am reading it wrong. Shortly about the project:
My ...
0
votes
0answers
163 views
Best platform for running (python and ( R or Octave)) algorithms for (large/big) data analytics [closed]
I have a machine learning algorithm currently implemented in R, wrapped in python (rpy2).
I would like to deploy this inside a web application and I am looking for the right platform to do this, ...
27
votes
8answers
2k views
Is sampling relevant in the time of 'big data'?
Or more so "will it be"? Big Data makes statistics and relevant knowledge all the more important but seems to underplay Sampling Theory.
I've seen this hype around 'Big Data' and can't help wonder ...
2
votes
3answers
359 views
Significance test for large sample sizes
This is more of a theoretical question. Super large sample sizes will almost always show a significance when a $\chi^2$ test is done. Is there any other statistical test of significance (an ...
2
votes
1answer
463 views
k-fold cross-validation strategy for large data set in statistical learning
I'm trying to learn the Bayesian network structure from a very large data set, and the R package I used for learning can only handle a very small portion of the data set (~10%) at one time due to the ...
6
votes
1answer
115 views
What are Effective Regression Techniques for Linguistic Analysis of Linked Data?
Cross-post from MathOverflow where it was suggested that I might get better results here.
I am in the early stages of a problem that involves parsing a large number ($\approx 5 \times 10^9$) of ...
1
vote
1answer
266 views
How to run survival analysis on big dataset?
I am recently involved in a project that needs to analyze the survival time of objects. Therefore, I plan to use the rms package to build a Cox model. The problem is, since the dataset I have is so ...
11
votes
0answers
305 views
First step for big data ($N = 10^{10}$, $p = 2000$)
Suppose you are analyzing a huge data set at the tune of billions of observations per day, where each observation has a couple thousand sparse and possibly redundant numerical and categorial ...
