0
$\begingroup$

Suppose I have a large data set(10 GB), with a response variable and multiple independent variables. What is the best way to utilize the data to build a model on?

If the full data set includes 10 million rows, would randomly sampling an arbitrary number(100,000 rows) and then modeling on that subset, be the best approach? How do you best utilize the full data set when you cannot use it due to memory and hardware constraints?

$\endgroup$
1
$\begingroup$

"Best" depends greatly on your research question. Resampling techniques, Monte Carlo simulations, and synthetic matching techniques are all very useful and are in line with the idea you mentioned.

Sometimes you can aggregate high frequency data into a more meaningful measurement, for example merging second by second data into 10 minute periods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.