Recently I've created a similarity measure specifically designed for high dimensional time series data that contain a low number of observation (measure) occurances. Within this, the equation is meant to accurately assess the relationship between non-linear variables despite the presence of time lag and high variance. These attributes are inherent, almost omnipresent details that often limit the exploration of time series in my emerging biological field; and thus data sets for validation purposes are hard to come by.

The problem: Validation. Niche field. Essentially, I have no gold standard for validation.

External Data sets: Few, if any external data sets resemble both the time series structure and contained variable profile targeted; and this is central as the relationship between time and the nature of my variables is largely driving the investigation. (ie. results from 15 minute intervals vs. common 1 hour intervals are completely different and answering different questions).

Literature: True positive control variables based on supported literature are not so common once the previously mentioned prerequisites are considered. But my search continues as this would allow me to use my own data for validation purposed.

Primary question (to be read as one question): How does one get around this issue? Is it possible to still validate? How can I empirically prove that my equation isn't specified for the benefit of my exact data, but is simultaneously specified and beneficial for this niche field? Any creative ideas on ways to validate within your own data to support a strong conclusion that the measure could be applied for use in future external data sets?

Of note: I thought about synthesizing data sets, but a few statisticians I trust commented that synthetically introducing lags would lead to bias, as time lag within my field is hardly studied, and that doing so would put my analysis at the predisposition that lags are important.

  • $\begingroup$ How did you get this data? Maybe you can get more from where this came from? $\endgroup$ – Antimony May 18 '17 at 19:16
  • $\begingroup$ The data originates from a project co-funded by a few universities; and is quite difficult to get more data as the data set cost is upwards near a million. I could try to scale down and generate another, but the human component and multiple time series measurements really ratchet up the cost and bureaucracy regardless. I'd prefer a 'simpler' approach if that exists $\endgroup$ – Aaron43 May 18 '17 at 19:25
  • $\begingroup$ Perhaps you can learn your model using the data and then test it on a synthetic data set? That way at the learning stage you are not introducing any synthetic biases. $\endgroup$ – Antimony May 18 '17 at 19:33
  • 1
    $\begingroup$ Good point. The dimensionality of my data typically creates problems in that process though, which ends up with me wondering around non-parametric cure-all fringe R packages deep in the night. I've had the thought that I could combine the 'importance score' to act as some sort of predictor-type centroid guiding data synthesis (but only a hoop dream so far). Also not sure if it clears me from the original problem of disproving the equation is only tailored for my data $\endgroup$ – Aaron43 May 18 '17 at 21:29
  • $\begingroup$ If your model performs poorly on the synthetic data, then it won't prove/disprove the original problem, but if it performs well, and on a variety of different synthetic datasets, I think one could say that it is able to generalize well to other datasets with similar characteristics. $\endgroup$ – Antimony May 18 '17 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.