I need to analyse a data set for my bachelor thesis and I am a little bit lost. The time series consist of measurements which were taken every 5th minutes from November 2016 until July 2017. Firstly I need to eliminate the negative values from the time series which are produced through malfunction in the measurement equipment. Then I need to find a model with which to eliminate double values which happen in a row. Lastly any exteme values ( For example when the previous intervals measurements have a value of 9,10,11,9,8,10 and then suddenly comes an interval with a value of 200 and the next one after is again 10) should also be counted as outliers.

Since I have not had a lot of experience with analysing time series, I would gladly hear suggestions how can I at best model the time series and with which program ?

Thank you in advance!

  • 1
    $\begingroup$ Your question raises many red flags for me, because all these procedures appear likely to bias any subsequent analyses. Could you explain why the measurements are integral, how negative values can arise, why double values need to be eliminated, and why unusually large values ought to be considered suspect? $\endgroup$ – whuber Aug 30 '17 at 12:40
  • $\begingroup$ These measurements are done with turbiditimeters in order to measure the turbidity of a river in nephelometric turbidity units (NTU). These turbiditimeters are installed in the river and measure continiously ( every 5th minute a new value). Negative values happen when the river runs dry and the turbiditimeter is not submerged anymore. These values have absolutely no use. Double values in a row are a measurement mistake from the turbiditimeter. These are decimal numbers, and when they are exactly the same in two 5-minutes intervals in a row, that's an error. $\endgroup$ – Delloman Aug 30 '17 at 13:45
  • $\begingroup$ Generally turbidity increases steadily when it rains and the water level of the river increases and that can be seen in the time series. When the weather is dry, turbidity is relatively constant.That means if there is a single really big value between a lot of small values when the weather is dry, this is quite possible an outlier. $\endgroup$ – Delloman Aug 30 '17 at 13:49
  • $\begingroup$ Are you saying that two readings are being reported for the same time point ? If so then you need to pre-process i.e. scrub your data to delete "doubles". Time series analysis i.e. developing an appropriate DGF ( data generating function) requires that anomalous values be neutralized so that they don't affect the analytics of model identification. These "unusal values" can be introduced into the forecast function via monte carlo methods reflecting the possibility that future values may contain anomalous effects. $\endgroup$ – IrishStat Aug 30 '17 at 14:01
  • $\begingroup$ No, only one reading is reported every 5 minutes. For example at 1:00 pm a value of 9,78433564 is recorded, at 1:05 pm a value of 10,378365334 is recorded, then at 1:10 pm the same value of 10,378365334 is recorded. This is most definitely an error from the turbiditimeter and should be considered an outlier. $\endgroup$ – Delloman Aug 30 '17 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.