2
$\begingroup$

I am working with a time series (discrete) that has ideally 1 value per time stamp. In some cases, we are seeing multiples that have a wide range all recorded with the same time stamp.

Up to now, we had been averaging the data but have found quite a few instances of very big ranges.

Should we instead be considering the modes (sometimes there are only 2 values), should we be considering a different method? When should we consider just throwing them out? There are many missing values in this data so using previous and following values in the sequence is not ideal.

I can't recall from my previous stats what the preferred method of dealing with these values is, as I haven't come across this in some time (maybe ever, in practice). Any ideas on how to handle the situation is greatly appreciated. Resources to look up would very much help as well!

Essentially the data sometimes looks like this, where the values are separated by ";" (these are not in sequence):

 00:43:00      "78;66;81"
 02:01:00      "68;74"
 03:53:00      "78;86;88;95;95;111;102;97;94;95"
 15:48:00      "81;120;97;58;84

Edit:

While the ideal is to have 1 per timestamp, the reason why there is more than one in some instances is the truncation at minutes by the database. However, we need to pick just one value to represent that timestamp as we don't care about the seconds details.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Please, add more details -- the crucial thing is WHY do you have multiple values if there should be one: are these accumulated over last timestep? Coincident in output from many detectors? Something else? $\endgroup$
    – user88
    Commented Mar 2, 2012 at 16:20
  • 1
    $\begingroup$ If the final ":00" is the seconds, which is truncated, you're saying you had 3 observations in one minute, then a gap of almost 80 minutes, then 2 observations in one minute, then a gap of over 100 minutes, then 10 observations in a minute, then a gap of over 700 minutes and then 5 observations in one minute? There has to be more structure to this problem than somewhat-irregular measurements truncated to the nearest minute. I could suggest using the median, but it seems like there's more to the story here. $\endgroup$
    – Wayne
    Commented Mar 2, 2012 at 18:16

1 Answer 1

Reset to default
1
$\begingroup$

(Sorry for submitting this as an answer instead of a comment, but I can't comment quite yet!)

If you can share any more information about the data I think it would help a lot in advising a solution.

What leads to there being multiple observations when "ideally" there should only be one? Is it a sensor (or data collection device) malfunctioning or some kind of human error?

Do you have any thoughts on why there is so much variation over what I am imaging to be a relatively small period of time? How long are the time intervals? If we are talking 1 measurement per hour and you happen to be getting two measurements in the hour, aside from me wondering why it is happening, you could also ask what time in the hour it is occuring (one at the start of the hour and one at the end of the hour, etc)? A potential solution if this is the case might be to take smaller time increments until you no longer have this problem, but that likely opens the door to a host of autocorrelation problems.

Does it lead to any periods with no observations (like the observation from period 1 just got shifted to period 2 and the observation from period 2 is still there)? Then you might just redefine period 1 and 2, though that might not work, depending on what exactly you are trying to do.

Essentially any information you can give to flesh out what exactly the problem is will really influence what advice I (or likely someone else with more real time series experience) will be able to offer you.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.