This is both a Math- and an R-question. I have a vector of POSIXct dates and a I want to determine the characteristics of the data.

Example with numbers:

c1 = 1,2,50,51,100,101,120,121
c2 = 10,20,30,40,50,60

c1 has more jumps and c2 is more continuous, but I am not sure how to sum this up in one value. Any ideas?

Consider a waiting queue, and the arrival dates as elements of the vector. Now I want to say: f(c1) < f(c2) for a function f to determine the smoothness of the arrival dates.

It is all about video uploads, in which I want to determine a value, whether there is a continuous frequency of uploads (daily upload) or a more bursty behaviour (5 videos on Monday and the next video on Friday)

  • 1
    $\begingroup$ Your reference to "jumps" suggests you are thinking in terms of the first differences of the data rather than the data themselves. The use of "more" suggests you are indifferent to the order, but are merely considering frequencies of differences. Why not then describe the distribution of first differences? $\endgroup$ – whuber Nov 3 '16 at 16:31
  • $\begingroup$ I'm not sure whether I understood you correctly ... The dates are sorted in ascending order but it does not matter when the jump appeares. It is all about video uploads, in which I want to determine a value, whether there is a continuous frequency of uploads (daily upload) or a more bursty behaviour (5 videos on Monday and the next video on Friday). $\endgroup$ – barracuda317 Nov 3 '16 at 16:43
  • 1
    $\begingroup$ That's exactly what I'm saying: by "jump" you appear to mean a difference in successive times and you don't care when it occurs. So please look at the distribution of time differences. $\endgroup$ – whuber Nov 3 '16 at 16:47
  • $\begingroup$ There's a good discussion of burstiness in this CV thread from 5 years ago. In particular, @nico's answer got high marks. stats.stackexchange.com/questions/12090/… $\endgroup$ – DJohnson Nov 3 '16 at 16:56

(This is an expansion on whuber's comment.)

Your data are arrival times, which are commonly summarized by the waiting time distribution (i.e. time difference between consecutive arrivals).

For example, a simple single-number summary might be the coefficient of variation of the arrival times. For the standard "memoryless" null model, the arrival time PDF would be exponential, with a unit coefficient of variation.

(This would be comparable to the "index of dispersion" approach discussed here, which applies to count data, # arrivals in a given time.)


+1 to @geomatt22 and whuber for their comments and suggestions. Their suggestions work when the information (jumps, first differences, etc.) is not extreme valued. A recent paper by Lin and Tegmark Critical Behavior from Deep Dynamics: A Hidden Dimension in Natural Language (available here ... https://ai2-s2-pdfs.s3.amazonaws.com/5ba0/3a03d844f10d7b4861d3b116818afe2b75f2.pdf), discusses situations that frequently occur where the information is power-lawed, extreme valued and exhibits critical complexity. In particular, they take "traditional" sequential analysis based on Markov processes to task for being "shallow" and unable to capture deep, long-term correlations. Here is their abstract:

We show that in many data sequences — from texts in different languages to melodies and genomes — the mutual information between two symbols decays roughly like a power law with the number of symbols in between the two. In contrast, we prove that Markov/hidden Markov processes generically exhibit exponential decay in their mutual information, which explains why natural languages are poorly approximated by Markov processes. We present a broad class of models that naturally reproduce this critical behavior. They all involve deep dynamics of a recursive nature, as can be approximately implemented by tree-like or recurrent deep neural networks. This model class captures the essence of probabilistic context-free grammars as well as recursive self-reproduction in physical phenomena such as turbulence and cosmological inflation. We derive an analytic formula for the asymptotic power law and elucidate our results in a statistical physics context: 1-dimensional “shallow” models (such as Markov models or regular grammars) will fail to model natural language, because they cannot exhibit criticality, whereas “deep” models with one or more “hidden” dimensions representing levels of abstraction or scale can potentially succeed.

This isn't intended to suggest that GeoMatt22 and Whuber's suggestions are wrong, it's merely intended to suggest an alternative formulation of the problem.

In addition to Tegmark's work, another recent paper by J.P. Bouchaud Crises and Collective Socio-Economic Phenomena (available here ... https://www.cfm.fr/assets/ResearchPapers/Crises+and+collective+socio-economic+phenomena.pdf) specifically discusses modeling behaviors such as sudden ruptures, crises and avalanches, which are close analogues to burstiness. Here is their abstract:

Financial and economic history is strewn with bubbles and crashes, booms and busts, crises and upheavals of all sorts. Understanding the origin of these events is arguably one of the most important problems in economic theory. In this paper, we review recent efforts to include heterogeneities and interactions in models of decision. We argue that the so-called Random Field Ising model (rfim) provides a unifying framework to account for many collective socio-economic phenomena that lead to sudden ruptures and crises. We discuss different models that can capture potentially destabilising self-referential feedback loops, induced either by herding, i.e. reference to peers, or trending, i.e. reference to the past, and that account for some of the phenomenology missing in the standard models. We discuss some empirically testable predictions of these models, for example robust signatures of rfim-like herding effects, or the logarithmic decay of spatial correlations of voting patterns. One of the most striking result, inspired by statistical physics methods, is that Adam Smith’s invisible hand can fail badly at solving simple coordination problems. We also insist on the issue of time-scales, that can be extremely long in some cases, and prevent socially optimal equilibria from being reached. As a theoretical challenge, the study of so-called “detailed-balance” violating decision rules is needed to decide whether conclusions based on current models (that all assume detailed-balance) are indeed robust and generic.

Together, these two papers represent significant advances in the analysis of extreme valued behaviors.

To be specific wrt your question about developing a "single" value to compare the two distributions, there are certainly many ways to do this. One way that incorporates the possibility of the information being extreme valued is to estimate the tail index of the distribution -- whether raw data, first differences, whatever. One easily generated approach to tail estimation is explained in Gabaix's paper on OLS modeling of the log-ranks (available here ... http://www.eco.uc3m.es/temp/jbes.2009.06157.pdf) or by leveraging the more rigorous and computationally intensive methods developed by Pickands or Hill. Once an index is available, then a distribution can be assigned based on the Examples section of this Wiki discussion of the Tweedie family of distributions... https://en.wikipedia.org/wiki/Tweedie_distribution

  • $\begingroup$ The quote "... mutual information between two symbols decays ..." to me indicates a slightly different situation from the OP('s problem, at least as it is stated in their question). That is, symbols indicates a situation where the "events" have attributes (besides just an arrival time). $\endgroup$ – GeoMatt22 Nov 3 '16 at 18:41
  • $\begingroup$ To me, there is sufficient generality in the OPs query to allow the introduction of this work into the discussion. $\endgroup$ – DJohnson Nov 3 '16 at 18:49
  • $\begingroup$ OK. I just think it is important, especially for inexperienced readers, to make distinctions clear. So, for example I do not think (?) you are talking about power-law waiting time distributions (e.g. Levy flight), but rather more similar to power law frequency-magnitude relations (e.g. in bulk and/or in time). $\endgroup$ – GeoMatt22 Nov 3 '16 at 19:02
  • $\begingroup$ To your point and to the best of my knowledge, the cited papers don't discuss queuing theory or waiting times. That may have been the example posited by the OP but does that limit the discussion to just that specific topic? I hope not. So, yes, frequency-magnitude relations are the gist of it. $\endgroup$ – DJohnson Nov 3 '16 at 19:15
  • $\begingroup$ I do not think your answer* is off topic per se, just that you may want to provide some more context for the OP. (*I would not really use the term "discussion" here. You should consider posting a separate question, to spark more discussion on the papers you mention. I would be interested. The abstracts read like a lot of "hand waving complex systems" papers I see in geophysics ... but deep learning has more results than earlier fads, so perhaps I will check them out!) $\endgroup$ – GeoMatt22 Nov 3 '16 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.