3
$\begingroup$

I'm fitting some machine learning algorithms (e.g. SVM) on my panel data. It's taking too long for my entire dataset, so I'm considering generating smaller samples from bootstrapping then fit the SVM in parallel.

At first glance, it may seem like by bootstrapping I will destroy the time-dimension of my data. However, if I include variable lags and moving averages, can I just bootstrap as normal (i.e. considering each observation as independent)?

Improve this question
$\endgroup$
4
  • $\begingroup$ Some sort of stochastic gradient descent would be the first direction I would turn if I had too much data, eg, scikit-learn.org/stable/modules/sgd.html. Although the question still remains over the preservation of block structure. Bootstrapping, in the sense of Efron , isn't going to help you to reduce computational effort because your resamples are the same size as the original data set. $\endgroup$
    – Andrew M
    Commented Nov 26, 2014 at 20:29
  • $\begingroup$ My thinking is to use bootstrap to generate smaller samples, which can be fit in parallel. Does this not help? $\endgroup$
    – Heisenberg
    Commented Nov 26, 2014 at 20:48
  • 2
    $\begingroup$ I'm not trying to be deliberately obtuse, but I find it confusing to call your procedure bootstrapping since that normally implies that the resamples are the same size as the original data set. I would just call it "subsample with replacement." (And it's not clear to me why this would be a preferred solution to subsampling without replacement.) $\endgroup$
    – Andrew M
    Commented Nov 26, 2014 at 21:49
  • $\begingroup$ That makes sense actually. Thanks for clearing up my thinking. $\endgroup$
    – Heisenberg
    Commented Nov 26, 2014 at 23:09

2 Answers 2

Reset to default
5
$\begingroup$

Generally, if you have a complicated data structure bootstrap sampling should reflect how real sampling was done. For example, with clustered or hierarchical data you rather sample whole clusters, than individual observations, because sampling individuals gives you a biased sample (e.g. Rena et al. 2010, Field and Welsh, 2007). For time-series two approaches were suggested in literature:

  1. Model-based resampling - you fit a model to the data to construct residuals from the fitted data and then generate new data using those residuals, so that you can sample "approximate disturbances" that can be found in your data.
  2. Block sampling - you sample blocks of data from time series (e.g. $x_{t-1}, x_t, x_{t+1}$) and then sample those blocks, so that local time structure is preserved.

You can find nice examples of those sampling strategies in a classic book by Efron, or in great handbook by Davison and Hinkley.

However, as also Andrew M mentioned, bootstrap applies to sampling with replacement $N$ observations out of $N$ observations, so this won't help with making the dataset smaller - it will make things worst, because you would have a sample of $N \times R$ draws.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm having block sampling in mind. My procedure is to sample observations independently, then for each selected observation also include the n-lags from the previous time periods. Is this procedure the same as block sampling? $\endgroup$
    – Heisenberg
    Commented Nov 26, 2014 at 19:22
  • 1
    $\begingroup$ In block sampling you sample whole blocks, not individuals, so the answer is not. Sampling individuals crushes the data structure. (See the inks provided) $\endgroup$
    – Tim
    Commented Nov 26, 2014 at 19:27
0
$\begingroup$

I would use something called cross-ssectional resampling discussed in this paper and here with less detail

I like this approach because its simple and intuitive. Basically where the panel data dimensions are time and individual unit, I would simply bootstrap (sub-sample) the individual units and keep all the observations for those units along the entire time period in the sample.

In other words, if you have $50$ individuals over $20$ time periods I would bootstrap (sub-sample) the $1:50$ individuals, and for those selected include all $1:20$ time periods.

The idea is that the data are likly auto-correlated in time, and you want to keep the time dependence structure intact while re-sampling, as to reflect the original sample.

If you for some reason had to reduce the size of the time dimension as well, then you would probably want to randomly sample by blocks of time.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.