2
$\begingroup$

I am trying to build a quantitative method for detecting that a multivariate dataset is in effect a time series, and for estimating its parameters. The Runs Test would be used for quantifying the non-randomness of the dataset's variables, and some measures of autocorrelation could be computed. But in order to move beyond this initial step, one would need to identify which variable in the dataset should be used as time dimension (if any). My question is the following:

How can one automatically detect this time dimension variable?

Of course, some simple taxonomy containing keywords like time, date, or timestamp could be used if variables are labeled. But this approach would require some localization, and could be difficult to implement if labels are not really explicit. For example, a non-native English speaker might not guess that DoB usually stands for Date of Birth, hence references a date.

If we consider that the dataset is already sorted in chronological order, one can assume that the variable that could be used as time dimension must have increasing values. If no matching variable can be found, one can assume that the dataset does not have any explicit time dimension, that all variables represent some kind of measurement, and that these measurements were recorded at regular intervals of unknown duration.

If a single matching variable can be found, it is quite likely to be the time dimension that we are looking for. But if multiple variables have increasing values, one might have values that are equally spaced. In which case, it should be picked as time dimension. Otherwise, we should consider that the dataset is some kind of log file which observations are event-driven rather than clock-driven.

This brings the real question at hand: assuming that we are looking at an event-driven log for which multiple variables have increasing values, which variable should be considered as best candidate for the time dimension? In other words, is there a context-independent way of ranking our candidate variables to suggest the one that has the highest probability of being the time dimension that we are looking for?

$\endgroup$
  • 1
    $\begingroup$ +1. From time to time I have had to do this sort of thing. I haven't the opportunity right now to explain, but one of the most powerful things you can do is compute the Fourier spectrum of each variable that has no duplicate values (and therefore appears to be a "continuous" variable rather than a discrete one): look for a single spike (and its harmonics). If all the rest can be treated as noise, you have a good candidate. The advantage of this somewhat indirect approach is that direct criteria, such as finding the GCD of all data differences, are ruined even by tiny amounts of imprecision. $\endgroup$ – whuber Dec 20 '14 at 1:01
  • $\begingroup$ What is the origin of the problem? Did you steal someone's log files and are trying to reverse engineer it? $\endgroup$ – Aksakal Dec 20 '14 at 2:25
  • $\begingroup$ No, I am trying to build an application that lets regular people who know very little about statistics analyze their datasets. You can follow our work on sutoiku.com. $\endgroup$ – Ismael Ghalimi Dec 20 '14 at 2:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.