I am writing some MapReduce code to calculate ordinary least squares from a sample of data. I'd like to include standard error, but I am running into a problem in calculating the variance of the noise. According to my reference, the variance is calculated by:

variance calculation

However, I don't compute $\hat\beta$ until the end. Thus, this estimate requires the entire sample set, which would defeat the point of MapReduce. If I wanted the standard error of the mean of a sample, for example, I could simply use an online variance calculation and aggregate the value until the end. Are there any alternatives for the OLS case where I could somehow aggregate a value throughout the job and use it at the end?

  • $\begingroup$ I think online least squares regression could be adapted. Would this be the sort of solution you are looking for? Alternatively, solutions for relatively large $p$ might be generated through sequences of independent ordinary LS problems via "matchers". Because MapReduce is so general and you provide no information about typical values of $n$ or $p$, it's hard to know which (if either) of these approaches to investigate further. Maybe you could tell us more about your specific problem? $\endgroup$ – whuber Nov 10 '14 at 21:03
  • 1
    $\begingroup$ I will look into that. In general I expect the use case for this to be "tall and skinny", so n will be large and p will be small. This is designed for business intelligence type users who have lots of data and several (but not a ton of) columns. $\endgroup$ – Nate Nov 10 '14 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.