1
$\begingroup$

I want to detect a change point in which a time series data changes its pattern. Ideally I want to draw a few red lines like shown in the below image.

I tried taking a log on y-axis, and running an offline bayesian change point detection with the methods shown here, but the results were not satisfactory. Im guessing that its because my data set is a hyperbolic data, which is non-stationary.

FYI, this is a production declining curve of oil & gas production. When everything is fine, the production data should follow a theoretical hyperbolic curve (after robust fit, which is the orange fitted line). But when there's something wrong on the well (such as well borehole gets clogged with mud or other fluids), production rate suddenly falls and stays that way until an engineer applies some "magic" on the well.

I want my algorithm to detect that new pattern. One thing that makes my problem difficult is that the severe downward outliers that are caused by temporary shut-down of well production (ex: for maintenance) may be identified as a change point. For example, in the bottom right plot, there are lots of downward outliers in the initial production phase due to temporary shut-down, but the production trend did not change (until 47th month).

How can I achieve this?

For your convenience, I provide the sample data points shown in the plots below.

enter image description here

data_1 = (5537.77, 9503.19, 5873.97, 7070.71, 6416.26, 5879.64, 3755.1, 3559.77, 2979.29, 2822.1, 1290.61, 2586.61, 2337.97, 2223.94, 2172.17, 2072.26, 2094.0, 1933.71, 1886.97, 937.6, 887.81, 836.73, 808.87, 791.71, 775.43, 756.29, 727.37, 720.9, 743.84, 733.29, 596.97, 612.63, 601.87, 610.55, 630.71, 672.2, 683.26)

data_2 = (28.03, 2536.77, 1854.55, 1477.14, 20.9, 1253.4, 16.77, 938.37, 948.84, 914.9, 814.03, 4.94, 731.2, 695.48, 9.1, 634.03, 443.1, 581.07, 7.71, 7.37, 7.23, 6.39, 317.4, 7.61, 470.8, 454.94, 445.45, 5.54, 5.35, 380.2, 424.97, 459.07, 354.84, 298.35, 366.47, 314.06, 376.77, 326.52, 4.32, 3.0, 3.71, 314.67, 3.84, 3.67, 5.94, 3.29, 3.27, 281.94, 287.9, 274.74, 267.68, 258.29, 272.06, 262.63, 254.26, 236.57, 230.03, 253.87, 268.73, 494.32, 477.43, 249.68, 207.23, 2.16, 173.43, 237.71, 265.93, 268.61, 264.16, 251.67, 251.74, 254.03, 255.35, 225.68, 475.18, 436.45, 428.53, 209.68, 256.13, 510.84, 474.29, 235.87, 191.65, 193.27, 169.32, 125.68, 104.43, 182.39, 110.9, 196.77, 57.57, 131.94, 255.37, 208.77, 190.43, 177.0, 179.94, 76.25, 194.39, 207.43, 130.16, 228.81, 204.26, 214.83, 201.23, 230.37)

data_3 = (372.81, 1484.23, 1326.32, 1248.29, 378.87, 1007.06, 1274.9, 775.06, 790.94, 1332.89, 174.39, 1264.37, 1381.52, 1316.1, 1306.61, 1273.71, 1288.23, 920.1, 1141.53, 1091.97, 1100.68, 1285.57, 1063.39, 1039.83, 1039.74, 957.93, 603.9, 919.42, 899.97, 832.48, 932.17, 872.0, 882.48, 868.76, 859.16, 852.97, 846.42, 837.83, 833.55, 833.71, 822.9, 809.48, 796.53, 780.61, 772.97, 763.82, 756.45, 751.8, 743.81, 743.83, 740.45, 728.97, 714.13, 713.74, 702.97, 687.68, 678.9, 663.82, 692.23, 598.4, 633.32, 624.2, 627.48, 636.29, 636.63, 623.45, 620.3, 599.84, 605.71, 588.43, 590.0, 584.6, 579.29, 582.83, 579.71, 546.9, 545.33, 546.32, 556.77, 561.94, 557.68, 412.38, 541.19, 528.1, 536.03, 530.2, 527.84, 525.65, 522.77, 519.71, 515.47, 513.48, 512.77, 480.0, 490.81, 474.8, 493.65, 487.33, 478.74, 477.97, 476.3, 64.42, 234.37, 479.74, 349.94, 267.68, 410.45, 361.7, 359.58, 358.6, 354.23, 320.29, 314.97, 276.71, 316.07, 333.16, 303.71, 283.29, 364.52, 356.03, 339.48, 314.4, 346.16, 344.81, 336.97, 326.32, 325.97)

data_4 = (1193.16, 3527.47, 2913.71, 2806.45, 1133.23, 2306.1, 1335.3, 346.68, 783.1, 745.0, 859.65, 801.87, 748.39, 922.5, 910.94, 795.52, 925.83, 944.9, 896.07, 892.23, 912.13, 812.18, 915.0, 898.37, 878.26, 768.43, 597.29, 754.9, 847.61, 800.14, 841.97, 712.83, 697.74, 688.73, 6.74, 656.93, 654.48, 614.35, 673.26, 667.25, 654.06, 660.23, 657.26, 649.67, 655.55, 652.29, 631.4, 632.16, 633.53, 618.32, 619.06, 582.71, 601.29, 614.77, 615.77, 599.67, 524.61, 1392.19, 1219.73, 1086.68, 1025.9, 969.48, 911.48, 892.71, 901.94, 862.03, 830.32, 806.7, 784.23, 752.97, 744.13, 726.61, 698.7, 676.55, 679.0, 651.62, 620.52, 637.87, 646.16, 635.1, 623.29, 612.13, 591.93, 616.26, 606.3, 585.1, 497.35, 557.0, 566.94, 469.83, 566.42, 542.53, 538.1, 536.03, 529.93, 71.68, 377.07, 492.26, 475.35, 342.61, 415.19, 451.0, 447.81, 410.67, 419.32, 399.58, 405.13, 399.39, 302.57, 302.48, 334.61, 330.29, 340.58, 392.17, 381.52, 382.17, 375.35, 369.97, 295.03, 404.03, 431.53)
$\endgroup$
2
  • $\begingroup$ A paper worth reading is this one: arxiv.org/abs/0710.3742 ... however, they are only carrying out the computations for the case where the data is normally distributed and the prior is the conjugate for the normal distribution (a crazy thing called inverse gamma distribution). For your case you would need to adapt it a bit but these computations you have to do anyhow I guess... $\endgroup$ Apr 19 '20 at 18:50
  • $\begingroup$ How were your results not satisfactory using the pymc package? Did you implement a hyperbolic version or did you just go with the intercept change? Did you draw the red lines manually? I think the red lines in the bottom right panel would be difficult to infer using a change point model. $\endgroup$ Apr 22 '20 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.