I'm dealing with about 4,000 stationary time series, most of which I was able to reasonably fit to a distribution based on the KS test. About 200 of the time series, however, were not so well-behaved and frequently consisted of the same value, repeated, and a few noticeable deviations from it.

For example: (200, 200, 200, 200, 200, 10, 200, 200, 200, 180, 200, 200, 200)

The numbers above are fictional but express the point (the data I'm using is proprietary and cannot be shared publicly). My question is: How do you model a process like that? I've been considering maybe some sort of switching mechanism, but with such few observations (each time series has about 50 observations, of which ~45 or more are repeated), any fit seems questionable at best. I also know the data are fundamentally sound after having spoken with the team that produces it -- so dropping these apparent "outliers" does not seem a viable option.

Please help! I've been struggling with this all weekend and am grasping at straws at this point!

Thank you,

  • $\begingroup$ I would first consider the possibility of data errors. I've seen data errors (temporarily off by 10x) in many time series, even ones I paid a high price to purchase. Then, I would get to understand the process that generates the data. Don't be like one professor I saw who spent two years to model something in the data... which turned out to be diagnostic values for a related process. $\endgroup$ – kurtosis Sep 1 '20 at 0:18
  • $\begingroup$ In this case, unfortunately, that is not the case. The actual values correspond to hydroelectric generation. By default, using the above values as an example, 200 MWh are generated, but every now and then an abnormally dry year comes along and causes lower-than-expected generation. $\endgroup$ – CasusBelli Sep 1 '20 at 14:38
  • 1
    $\begingroup$ Ah, very helpful to know! $\endgroup$ – kurtosis Sep 1 '20 at 19:48

I think there are a few ways you could look at your data. On the one hand, you have a dataset that is almost always right-censored. You could try to model the maximum MWh output if the plant were not limited, but that would be very tough with the data you have (small number of observations, 2 of 13 being right-censored).

Probably easier would be to assume that most years the plant will have sufficient head to generate 200 MWh and model the situations where it does not. I'd say the insufficient head periods could be arriving at some Poisson rate with some marker on those arrivals for how much below capacity it will be.

You could also try something like a survival analysis. Your problem is like modeling lifetimes for a disease where most monitoring stops after some number of years. That captures the left and right censoring (constraints to lie between 0 and 200), but it may be a bit inaccurate on the downside.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.