Doing regressions on samples from a very large file: are the means and SEs of the sample coefficients consistent estimators? I have a fairly larege file 100M rows and 30 columns or so on which I would like to run multiple regressions. I have specialized code to run the regressions on the entire file, but what I would like to do is draw random samples from the file and run them in R.
The strategy is:
              randomly sample N rows from the file without replacement
              run a regression and save the coefficients of interest
              repeat this process M times with different samples 
              for each coefficient calculate the means and standard errors  of 
                the coefficeints over M runs.
I would like to interpret the mean computed over M runs as an estimate of the values of the coefficients computed on the whole data set, and the stadard errors of the means as estimates of the standard errors of the coefficients computed on the entire data set.
Experiments show this to be a promising strategy, but I am not sure about the underlying theory. Are my estimators consistent efficient and unbiased? If they are consistent how quickly should they converge? What tradeoffs of M and N are best?
I would very much appreciate it if someone could point me to the papers, books etc. with the relevanth theory.
Best regards and many thanks,
Joe Rickert
 A: If you can assume that your rows of your data matrix are exchangeable then your modelling strategy should work well. Your method should be fine under the conditions stated by Gaetan Lion before. 
The reason why your method will work (given the exchangeability assumption holds) is that it be taken as a special case of parametric bootstrap in which you take re-sample N rows of big sample, fit a model and store the coefficients and repeat this M times (in traditional bootstrap terminology your M is equivalent to B) and take average of the M coefficient estimates. You can also look at it from a permutation testing view point as well.
But all these results are true if the (hard to verify) exchangeability assumption holds. If exchangeability assumption doesn't hold, the answer in that case becomes a bit complicated. Probably you need to take care of the subgroups in your data which are exchangeable and perform your process conditioned on these subgroups. Basically, hierarchical modeling.
A: The answer to your original question is yes, because the classical theory applies under your sampling scheme.  You don’t need any assumptions on the original data matrix.  All of the randomness (implicitly behind standard errors and consistency) comes from your scheme for sampling $N$ rows from the data matrix.
Think of your entire dataset (100M rows) as being the population.  Each estimate (assuming your sample of size $N$ is a simple random sample of the rows) is a consistent estimate of the regression coefficients (say, $\hat{\beta}_*$) computed from the entire data set.  Moreover, it is approximately Normal with mean equal to $\hat{\beta}_*$ and some covariance.  The usual estimate of the covariance of the estimate is also consistent.  If you repeat this $M$ times and average those $M$ estimates, then the resulting estimate (say, $\hat{\beta}_{avg}$) will also be approximately Normal.  You can treat those $M$ estimates as being nearly independent (uncorrelated) as long as $N$ and $M$ are small relative to 100M.  That’s an important assumption.  The idea being that sampling without replacement is approximately the same as sampling with replacement when the sample size is small compared to the population size.
That being said, I think that your problem really is one of how to efficiently approximate the regression estimate ($\hat{\beta}_*$) computed from the entire data set.  There is a difference between (1) averaging $M$ estimates based on samples of size $N$ and (2) one estimate based on a sample of size $MN$.  The MSE of (2) will generally be smaller than the MSE of (1).  They would only be equal if the estimate was linear in the data, but that is not the case.  I assume you are using least squares.  The least squares estimate is linear in the $Y$ (response) vector, but not the $X$ (covariates) matrix.  You are randomly sampling $Y$ and $X$.
(1) and (2) are both simple schemes, but not necessarily efficient. (Though it may not matter since you only have 30 variables.)  There are better ways.  Here is one example: http://arxiv.org/abs/0710.1435
A: The greater the sample N, the smaller the standard error (higher t stat, and smaller the respective p values) associated with all your regression coefficients.  The greater M, the more datapoints you will have and the smaller will be your standard error of the mean of the coefficients over M runs.  Such means should have a standard error that is normally distributed per the Central Limit Theorem.  In terms of convergence of such means, I am not sure there are any statistical principles that dictate this.  I suspect if your random sampling is well done (no structural bias, etc...) the convergence should occur fairly rapidly.  That is something you just may have to observe empirically.
Otherwise, your method seems good, I don't see any problem with it.   
