I'm currently working on analysing data from a simulation. The simulation result are statistical twins, which only vary in one point, the usage of an application to find a fueling station, that is cheap. In advance I have to say that my statistic skills are a little rusty.

I want to measure the effect of such an application on the fuel price. To do that me and my advisor at the university decided to use a linear regression. I came up with the following formula: $$ Y_{€⁄L}=β_0+β_1 X_{App}+β_2 X_{Oilprice} +β_3 X_{hour}+β_4 X_{day}+β_5 X_{fueltype}+ β_6 X_{highwaystation}+β_7 X_{brand}+ Z_{Home}+ Z_{Work}+ Z_{Station}+ϵ $$ The $Z_{work}$, as well as the others, represent a geometry within germany, that is identified by something call Regionalschlüssel, which is a unique number like 03255. The reason why I want to include this kind of information is, that this way I can observe the effects on the fuelprice that might be caused by local price differences. (I really hope I understood that right)

My current linear regression in R looks like lm1 <- lm(price ~ app + oilprice + hour + day + fuel_type + brand + factor(start_rs) + factor(end_rs) + factor(station), data=queried_data)

The problem with this is, that my machine runs out of memory due to the very high amount of observations (9.759.911).

Error: cannot allocate vector of size 85.5 Gb
In addition: Warning messages:
1: In model.matrix.default(mt, mf, contrasts) :
  Reached total allocation of 16005Mb: see help(memory.size)
2: In model.matrix.default(mt, mf, contrasts) :
  Reached total allocation of 16005Mb: see help(memory.size)
3: In model.matrix.default(mt, mf, contrasts) :
  Reached total allocation of 16005Mb: see help(memory.size)
4: In model.matrix.default(mt, mf, contrasts) :
  Reached total allocation of 16005Mb: see help(memory.size)

I have already done some reseach on other packages like

  • lfe
  • lme4

but as I said in the beginning my skillset is not high enough to comprehend them, nor is my english.

It would really help me if you could point me in the right directions.

  • $\begingroup$ Is there a reason you have chosen to simulate 9,759,911 observations? What would be lost if you stopped at, say, 10,000 observations? $\endgroup$ – whuber Sep 7 '15 at 20:17
  • $\begingroup$ The results of the simulation are 9.759.911 observations. If I stop at 10.000 I would only observe a very very small amount ~80 of 370,000 simulated commuters. $\endgroup$ – Benjamin Sep 7 '15 at 22:23
  • $\begingroup$ Let me be a little more blunt: why do you need so many simulated results? In the vast majority of applications people can obtain the information they need with far less data. What is it about your situation that compels you to perform such an extensive simulation? What are your options for generating less data or subsampling the data you have? $\endgroup$ – whuber Sep 8 '15 at 13:31
  • $\begingroup$ @whuber you were right, I was able to get similar results with a smaller sample of my data, but I was also able to do the regression with the full data set. $\endgroup$ – Benjamin Sep 15 '15 at 9:26

Taking whubers comments in consideration, I pick a representive sample from my 9mio oberservations and did the linear regression with this smaller sample and it worked just fine.

f.day. <- as.factor(obs.sample.factors$zeday); f.day. <- relevel(f.day., ref="Mon")
f.time. <- as.factor(x=obs.sample.factors$time_slotted); f.time. <- relevel(f.time., ref="morning")
f.station. <- as.factor(obs.sample.factors$rs_station)
fit.sample <- lm(price ~ app + bab_station + brand + oilprice + fuel_type + f.time. + f.day. + f.station., data=obs.sample.factors)

But I still wanted to know the results if I did the regression with the full data set. On CRAN there is a package called lfe which was made to handle such big data sets. The downhand of this package for me was, that I was not able to do tests for heteroskedastisity etc. (I'm just not a statistician nor R specialist). But I was able to use the full dataset with the following R code

ft.day <- as.factor(obs.total$zeday); ft.day <- relevel(ft.day, ref="Mon")
ft.time <- as.factor(x=obs.total$time_slotted); ft.time <- relevel(ft.time, ref="morning")
ft.station <- as.factor(obs.total$rs_station)
fit.total <- felm(price ~ app + bab_station + brand + oilprice + fuel_type | ft.time + ft.day + ft.station, data=obs.total)

I hope this will help out anyone trying to do a linear regression with huge datasets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.