When looking at papers about long memory they tend to analyze data sets whose length is in the thousands, see http://www.math.canterbury.ac.nz/~m.reale/pub/Reaetal2011.pdf for an example.

My question is to the long memory researchers and practitioners out there. What rule of thumb do you use to decide whether a data set is too small to be able to appropriately detect/estimate long memory?

[Naturally the smaller the long memory parameter the more observations you will require to detect it, but i'm after a general rule of thumb rather than exact notions around the number of observations required to detect a specific effect size.]

  • 3
    $\begingroup$ One thing you might want to try is lookup the $n$ used in simulation studies in a few theory papers. If you find that they generally use a sample size over a certain value, you can probably assume that for $n$ smaller than that there is decent power loss. $\endgroup$ Jun 12 at 22:27
  • $\begingroup$ The problem is that what happens in simulation studies and how models behave in practice are very different things. My experience is that people choose round numbers that work and/or reviewers ask for longer/large data sets. It could provide an upper-lower bound though. Thanks for the suggestion! $\endgroup$
    – adunaic
    Jun 14 at 0:43
  • $\begingroup$ There is no hard or soft $n$. My rule of thumb is how much better can we do compared to a simple model. An OLS $y = \beta_0 + \beta_1t$ and then add the world's simplest periodic term so: $y= \dots + \beta_2sin(2\pi\frac{t}{T})+ \beta_3cos(2\pi\frac{t}{T})$ if there is obvious periodicity $T$. Takes ~15' to set up the data for - maybe throw an AR(1) error term if I think it can get estimated. In that sense, the question is on some false premises. There is no reason to use a GBM if a simple binary tree provides adequate performance or the GBM provides inadequate gains. Same thing with the LSTM. $\endgroup$
    – usεr11852
    Jun 15 at 19:54
  • $\begingroup$ I appreciate the premise that simpler models can often perform very well but I can't see how this comment is relevant to the question at hand. $\endgroup$
    – adunaic
    Jun 16 at 10:16


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.