How to detect and quantify a structural break in time-series (R)

Question

Background

So first some background to gauge the level of understanding I might have. Currently completing MSc thesis, statistics has been a negligible part of this although I do have a basic understanding. My current question makes me doubt what I can/should do in practice, reading more and more online and in literature seems to be counterproductive.

What am I trying to achieve?

So for my thesis I joined a company and the general question I am trying to answer there is essentially how a predictive process is affected by the implementation of certain system (which affects the data used for the predictive process).

The desired outcome in this is an understanding of:

1. Is there a noticeable change? (e.g. statistical proof)
2. How large is the change? (in mean and variance)
3. What factors are important in this predictive process (Also how the influence of factors changes from before > after the break)

To answer 1 and 2 I obtained historical data in the form of a time-series object (and more but irrelevant at this stage). The software I use is R.

Data

The data encompasses a weighted score for each day (2.5yrs), indicating how bad the predictive process performed (deviation from the actual event). This one time-series object contains the weighted score for the predictions that occurred from one hour before up until the actual occurrence of the event (1hr interval) for these 2.5 years (so each day has one weighted score for this interval). Likewise, there are multiple time series constructed for other intervals (e.g. 1-2, 2-3 hrs etc.)

myts1 <- structure(c(412.028462047, 468.938224875, 372.353242472, 662.26844965, 
                 526.872020535, 396.434818388, 515.597528222, 536.940884418, 642.878650146, 
                 458.935314286, 544.096691918, 544.378838523, 486.854043968, 478.952935122, 
                 533.171083451, 507.543369365, 475.992539251, 411.626822157, 574.256785085, 
                 489.424743512, 558.03917366, 488.892234577, 1081.570101272, 488.410996801, 
                 420.058151274, 548.43547725, 759.563191992, 699.857042552, 505.546581256, 
                 2399.735167563, 959.058553387, 565.776425823, 794.327364085, 
                 1060.096712241, 636.011672603, 592.842508666, 643.576323635, 
                 639.649884944, 420.788373053, 506.948276856, 503.484363746, 466.642585817, 
                 554.521681602, 578.44355769, 589.29487224, 636.837396631, 647.548662447, 
                 740.222655163, 391.545826142, 537.551842222, 908.940523615, 590.446686171, 
                 543.002925217, 1406.486794264, 1007.596435757, 617.098818856, 
                 633.848676718, 576.040175894, 881.49475483, 687.276105325, 628.977801859, 
                 1398.136047241, 749.644445942, 639.958039461, 649.265606673, 
                 645.57852203, 577.862446744, 663.218073256, 593.034544803, 672.096591437, 
                 544.776355324, 720.242877214, 824.963939263, 596.581822515, 885.215989867, 
                 693.456405627, 552.170633931, 618.855329732, 1030.291011295, 
                 615.889921256, 799.498196448, 570.398558528, 680.670975027, 563.404802085, 
                 494.790365745, 756.684436338, 523.051238729, 535.502475619, 520.8344231, 
                 623.971011973, 928.274580287, 639.702434094, 583.234364572, 623.144865566, 
                 673.342687695, 567.501447619, 602.473664361, 655.181508321, 593.662768316, 
                 617.830786992, 652.461315007, 496.505155747, 550.24687917, 588.952116381, 
                 456.603281447, 425.963966309, 454.729462342, 487.22846023, 613.269432488, 
                 474.916140657, 505.93051487, 536.401546008, 555.824475073, 509.429036303, 
                 632.232746263, 677.102831732, 506.605957979, 701.99882145, 499.770942819, 
                 555.599224002, 557.634152694, 448.693828549, 661.921921922, 447.00540349, 
                 561.194112634, 590.797954608, 590.739061378, 445.949400588, 725.589882976, 
                 480.650749378, 587.03144903, 483.054524693, 428.813155209, 540.609606719, 
                 495.756149832, 409.713220791, 492.43287131, 618.492643291, 723.203623076, 
                 461.433833742, 420.414959481, 480.501175081, 564.955582744, 453.0704893, 
                 506.711353939, 521.12661934, 487.509966405, 483.442305774, 506.932771141, 
                 442.871555249, 873.285819221, 1201.628963682, 1392.479592817, 
                 693.292446258, 629.477998542, 660.777526646, 414.376675251, 475.517946081, 
                 501.626384564, 470.216781646, 444.195433559, 697.258566625, 546.966755779, 
                 428.945521943, 388.203080434, 579.759476551, 548.433130604, 453.950530959, 
                 460.613845164, 534.329569431, 560.663080722, 660.799405665, 432.3134958, 
                 569.59842379, 518.195281689, 650.007266105, 521.642137647, 442.763872575, 
                 687.470213886, 951.651918891, 589.611971045, 493.203713291, 431.966577408, 
                 616.912296912, 685.80916291, 502.518373775, 595.630289879, 563.104035749, 
                 523.383707347, 532.042896625, 470.949823756, 603.408124923, 615.301428799, 
                 708.26541245, 725.853182875, 705.777543119, 530.351781147, 698.828825921, 
                 462.173187592, 366.411986505, 848.613888761, 502.940599188, 456.044881766, 
                 605.321231272, 629.861109863, 431.130428123, 509.672767868, 457.598828697, 
                 553.932034119, 610.181457495, 581.59017099, 540.788638119, 705.226962669, 
                 610.670142045, 566.392016015, 611.086310256, 603.256299175, 766.372982953, 
                 801.921868916, 761.708239486, 580.712445849, 575.53616943, 540.066255921, 
                 608.133122153, 735.063468208, 637.091441112, 778.874033589, 689.350099602, 
                 1003.219851026, 624.107808848, 635.887051641, 420.915060155, 
                 511.460563095, 817.08209288, 603.089908306, 772.6493477, 797.148459813, 
                 588.255963229, 499.050860875, 502.059987, 565.524637543, 1663.182976069, 
                 2281.49950544, 1442.687607103, 1024.355834401, 899.519857882, 
                 988.585993922, 612.834835776, 641.686600038, 717.951451466, 746.441686309, 
                 1147.770724052, 596.279691286, 932.861076555, 497.228997645, 
                 764.895725484, 659.054003787, 1148.227820587, 1403.462969143, 
                 624.733620842, 803.199038618, 839.637983048, 1278.286165347, 
                 774.363457936, 662.767213211, 627.251799204, 650.180035442, 1296.405174964, 
                 662.928010153, 523.095967567, 620.727894789, 650.876097695, 509.534317267, 
                 479.922326477, 613.743251306, 430.117763379, 1825.108688714, 
                 744.708270099, 455.818978039, 370.908485795, 771.317824437, 688.219350724, 
                 468.16351523, 791.649828808, 666.360829114, 1427.809117119, 2861.163543428, 
                 1090.887950582, 621.942045727, 397.381382335, 397.697308586, 
                 494.441558442, 474.314526966, 888.812606506, 476.031636688, 651.907747324, 
                 389.95997873, 680.776897408, 1499.093314237, 1077.571595752, 
                 765.690897368, 571.545469449, 590.64855754, 492.371592484, 580.811781306, 
                 873.628734717, 602.958435426, 549.877214337, 546.66120979, 394.75285753, 
                 520.238244635, 517.217468365, 903.057976974, 528.477241796, 378.958677302, 
                 491.589659729, 548.665964908, 453.512746452, 481.081050678, 491.499714029, 
                 628.539705456, 672.540312912, 1686.825394554, 1367.577856001, 
                 600.373039737, 417.511405109, 511.75535978, 440.677427555, 493.430816323, 
                 533.025975459, 547.429120615, 432.168874608, 555.098163047, 521.644301834, 
                 667.159371501, 421.591007887, 757.218378664, 615.572602597, 433.961482908, 
                 528.813953729, 633.228715271, 519.648748842, 437.342815473, 551.877832301, 
                 703.377801948, 536.673383258, 658.597165739, 1449.850501569, 
                 615.204142853, 499.197033946, 853.692014263, 490.213941347, 812.68824521, 
                 521.364349414, 818.757704456, 848.59674442, 646.819554339, 471.051626838, 
                 598.326620222, 782.58569568, 754.880939869, 636.572395084, 686.076138643, 
                 530.158582782, 524.696479569, 525.441231521, 593.834663615, 415.830854949, 
                 590.135594493, 591.019407595, 503.321975981, 515.371205208, 494.805384342, 
                 567.397190671, 482.180658052, 724.099533838, 791.107121538, 564.673191002, 
                 572.551388184, 729.46937136, 943.538757014, 519.051645932, 994.190842696, 
                 866.69659257, 610.021553913, 547.791568399, 578.854543644, 684.826681706, 
                 815.179238308, 617.050464226, 623.818649573, 537.163825262, 529.850027242, 
                 926.531531345, 588.578930644, 457.329084489, 380.160216157, 494.287689357, 
                 463.885244047, 451.611520014, 762.508948042, 773.74942889, 1642.691010358, 
                 555.226392541, 659.433830806, 454.348720108, 388.274823265, 650.63824747, 
                 632.327400443, 584.93699748, 484.815917524, 733.153950316, 471.349864174, 
                 418.755413722, 547.060192029, 742.028289483, 521.119798289, 1176.207996336, 
                 524.730544122, 430.009783422, 558.479383664, 574.162550914, 526.08247269, 
                 611.207728202, 551.202548069, 472.046973518, 517.490179087, 556.135143079, 
                 628.084374004, 413.677676623, 439.814082201, 1011.775306843, 
                 684.443831473, 546.421742134, 578.853727684, 517.693483714, 638.112468944, 
                 631.531739664, 501.897019514, 661.11860926, 521.695715961, 474.403897254, 
                 463.294645328, 559.583511974, 531.953658919, 740.412596176, 534.815607516, 
                 462.329096628, 637.941748843, 702.69170843, 471.390065606, 590.458408612, 
                 617.006573387, 565.411288964, 472.986933034, 567.745850996, 596.925622448, 
                 474.068038429, 653.56453828, 612.893376781, 711.545758298, 527.783301631, 
                 478.530081662, 519.751192408, 536.550807025, 443.437342694, 587.403769673, 
                 601.15805729, 556.497167238, 374.228230116, 477.027420471, 494.984999444, 
                 879.314339401, 704.997313272, 626.546803934, 653.296523326, 435.581408863, 
                 633.048339362, 403.889616794, 488.214190958, 575.631003993, 430.984422675, 
                 437.83561603, 522.277281965, 475.602597701, 527.12160277, 944.139469794, 
                 474.50403295, 579.478722386, 459.088134733, 503.246692031, 610.022771263, 
                 446.143895372, 625.022916127, 517.435543013, 891.375454252, 555.864115385, 
                 474.764739145, 921.714956231, 645.896256587, 1536.221634415, 
                 816.575921465, 596.491670621, 503.56011064, 720.743463226, 905.835642175, 
                 1360.481537034, 653.224092421, 633.505228314, 546.064475635, 
                 482.454025258, 962.715357696, 618.202090733, 803.895156435, 668.047995992, 
                 594.566585046, 839.597813143, 457.375793588, 631.863607862, 475.266615122, 
                 664.569635822, 481.886574644, 1614.962054217, 869.212340286, 
                 501.400781534, 478.670649186, 521.824073342, 684.720851031, 597.124676952, 
                 605.903108456, 491.358096619, 430.812042311, 388.350092055, 488.132638097, 
                 413.131448595, 391.891460495, 430.760685279, 731.99097305, 382.200799877, 
                 511.48361093, 560.620999712, 528.369543055, 536.348770159, 721.297750609, 
                 491.321646454, 509.521489714, 561.318889907, 553.24041301, 459.235996646, 
                 354.741174128, 339.775552834, 432.548724483, 438.672630955, 508.177204773, 
                 496.199702536, 643.867549669, 611.460979278, 861.190516859, 662.56052508, 
                 524.398593443, 529.585928069, 607.575374022, 495.001029442, 700.371352785, 
                 794.753142167, 466.792229932, 435.426320832, 450.903747896, 622.562955777, 
                 1562.215153595, 725.069249874, 612.357398912, 418.579228487, 
                 381.667629501, 528.173266471, 687.876352966, 655.845568131, 423.589678964, 
                 612.545707971, 951.362478322, 1800.162370822, 600.672989388, 
                 531.048286916, 527.565406977, 402.380659606, 607.699770367, 1486.296473731, 
                 686.560841226, 4176.136413427, 3086.067140966, 1872.815975088, 
                 771.413460362, 843.791946967, 652.825527602, 642.443948966, 726.208291336, 
                 641.092848676, 488.237988698, 606.154989706, 1426.027951807, 
                 959.347533388, 649.856202928, 527.580884911, 400.545393834, 568.268813107, 
                 631.257023117, 515.755741256, 682.375587555, 583.855170876, 506.146152757, 
                 517.095094378, 563.415777949, 801.015579658, 649.56360904, 732.097267107, 
                 456.626323752, 499.170138889, 549.393587002, 556.589070013, 590.180621262, 
                 667.709332802, 421.738377899, 661.178862228, 570.833727593, 631.139001868, 
                 545.835879493, 559.918523671, 1364.379214546, 985.777069008, 
                 644.949427255, 493.066294248, 476.852498787, 379.716401582, 715.333935018, 
                 459.326945313, 621.665546323, 476.317803131, 519.803138696, 409.241665463, 
                 465.206511176, 594.689036224, 443.841857849, 399.830019307, 570.65982956, 
                 516.562325113, 381.909941529, 532.130831616, 650.329631588, 661.055942562, 
                 1136.942413908, 508.543555485, 976.852889691, 1461.16921717, 
                 646.062436059, 593.093537367, 624.839875084, 453.453385269, 584.633165187, 
                 507.616009915, 516.857276979, 434.651983821, 572.755844368, 454.901132196, 
                 707.698546138, 760.341584614, 449.252091224, 623.217222998, 625.061550699, 
                 2030.045687713, 1582.036383383, 677.325281969, 571.588930686, 
                 493.235172445, 556.291968991, 424.360693057, 436.333980583, 484.105667103, 
                 505.231040152, 378.767240615, 495.943549377, 321.856525703, 363.651848067, 
                 557.201599565, 603.658298878, 558.958198405, 789.717963533, 480.370977054, 
                 509.366153138, 467.526623793, 576.508422894, 661.322171003, 520.804998847, 
                 342.109381368, 473.512224982, 984.139466992, 487.586712759, 605.914245454, 
                 459.190981983, 678.728907858, 342.511103348, 436.746013478, 520.896987467, 
                 818.078350515, 527.494249096, 713.52499017, 610.365469264, 462.965548015, 
                 362.931986459, 810.610193032, 393.455578799, 536.720944152, 551.490260933, 
                 464.369987186, 275.832746918, 513.723009815, 491.945195301, 438.865839297, 
                 257.252871794, 615.513481211, 420.507536576, 392.035094971, 392.963333027, 
                 435.276624468, 253.431425091, 592.873595776, 500.615067792, 503.491101855, 
                 475.352827724, 1135.11762886, 723.666909467, 712.259187274, 559.738346197, 
                 490.958692763, 435.998397207, 729.341315271, 406.369683231, 632.626098862, 
                 565.318329487, 394.031553179, 356.627786519, 374.075606064, 336.505546227, 
                 393.168901965, 480.183256037, 573.840777708, 187.680483645, 170.978544639, 
                 209.134883957, 193.039610198, 224.362544607, 210.946012575, 166.006351727, 
                 201.500604051, 160.008039339, 229.847327915, 193.655724693, 255.575881835, 
                 207.0547762, 186.835573846, 247.776981217, 574.489201771, 602.683083111, 
                 202.592131296, 205.827635803, 279.139480224, 367.898232724, 245.75114848, 
                 304.004111775, 231.706217872, 159.750034155, 207.584399879, 239.854118732, 
                 212.711257914, 181.755353255, 225.98761652, 206.570825973, 193.940523176, 
                 203.851963746, 170.700301226, 226.41517367, 183.87542452, 206.640800387, 
                 218.953856594, 176.127748987, 168.205878516, 201.718224491, 171.421791143, 
                 193.065350731, 235.118975292, 217.002358158, 197.563916607, 219.266588546, 
                 168.044251777, 182.1752633, 172.514392523, 192.250197416, 171.023249603, 
                 151.848998035, 151.678097242, 173.28609604, 194.852548067, 178.742929426, 
                 180.591580951, 189.241987711, 206.960325123, 207.639468123, 307.389246296, 
                 197.203559504, 201.143394951, 313.861245405, 215.87226561, 218.500147257, 
                 201.993836847, 204.461813487, 180.04803872, 305.301279156, 214.880173012, 
                 250.930811019, 284.90370816, 212.518096229, 195.034955147, 190.236951118, 
                 257.088555132, 189.502428671, 239.652450054, 243.53269907, 289.820554649, 
                 249.011846483, 279.393483142, 194.006529415, 214.59905786, 247.746491522, 
                 224.723467084, 191.086994265, 271.654470268, 183.455652269, 327.307813653, 
                 149.721791725, 170.538132352, 192.954323882, 172.459631548, 199.829129152, 
                 163.257916856, 179.33550074, 207.618752518, 191.969484656, 191.506188821, 
                 176.446694609, 194.818608621, 167.891653012, 167.000377869, 201.869893803, 
                 183.895614224, 148.510174938, 157.27033831, 166.573696008, 309.557485177, 
                 595.559287832, 232.318746483, 188.295593138, 157.721093739, 177.753625321, 
                 214.844940799, 226.812056373, 213.910815993, 188.973729833, 243.306529363, 
                 202.733401293, 197.296870477, 182.523989567, 170.428625204, 172.593507241, 
                 151.926342178, 283.415923815, 163.456408737, 172.273699114, 159.382452668, 
                 182.158974043, 156.532047405, 182.768725504, 239.318216814, 151.579013694, 
                 148.216885776, 241.346766195, 257.110058666, 147.931152011, 186.2508302, 
                 197.379372605, 155.69798218, 147.370235658, 180.961415272, 168.372900875, 
                 297.9454882, 177.640856836, 170.535749604, 166.211309731, 178.463064513, 
                 175.035906056, 179.971897406, 162.858094293, 164.056675362, 250.151933362, 
                 195.49943188, 175.281720263, 182.392087534, 144.341091392, 241.466298465, 
                 184.806150673, 167.932448156, 150.621075732, 178.510315062, 209.913152992, 
                 222.798716921, 153.536476868, 190.326281004, 173.708412622, 148.075945402, 
                 149.147879365, 181.5851776, 141.012604897, 157.982751461, 159.364653673, 
                 151.777006477, 191.065323099, 139.076429662, 172.452344359, 182.482693518, 
                 287.424045314, 169.327943406, 298.970852166, 145.238645275, 175.211717467, 
                 165.116950464, 187.006564993, 160.959119092, 198.552333833, 170.406040157, 
                 162.10770001, 146.356131036, 170.336552623, 163.095730104, 155.192077125, 
                 217.182324478, 231.270198833, 256.905250226, 184.59676245, 175.557146633, 
                 164.246298131, 221.695058452, 197.911691457, 188.427830442, 259.361745153, 
                 164.243672823, 190.67188784, 182.331604811, 190.352555581, 248.738493256, 
                 196.854564795, 164.974185334, 332.650385373, 169.081552611, 193.578840033, 
                 192.166911863, 214.174943222, 271.287900593, 224.675083031, 171.950208574, 
                 173.867031268, 139.260432794, 177.012491325, 171.268066406, 132.714578168, 
                 197.224558817, 152.561299656, 143.415562042), .Tsp = c(2016.3306010929, 
                                                                        2018.99909424358, 365), class = "ts")

Process until now

Now I had understood that for question 1 I can apply a test for a structural break to determine if and when the break occurred (with a known break date). For this I use strucchange package in R and utilize the breakpoints function.

However, the CUSUM (for unkown break date) test was also recommended by my supervisor. Unsure what is best here?

EDIT:

I see Andrew's supF test conducts the Chow's test for all possible breaks. Then rejects if the maximum of the F (or Chow) statistics become too large. (Found - perform chow test on time series)

Code to obtain a break date using struccchange

library(strucchange)
test2 <- Fstats(myts1~1) #Gets a sequence of fstatistics for all possible 
# break points within the middle 70% of myts1
myts1.fs <- test2$Fstats #These are the fstats
bp.myts1 <- breakpoints(myts1~1) #Gets the breakpoint based on the F-stats
plot(myts1) #plots the series myts1
lines(bp.myts1) #plots the break date implied by the sup F test
bd.myts1 <- breakdates(bp.myts1) #Obtains the implied break data (2018.35, 
# referring to day 128 (0.35*365 = day number))
sctest(test2) #Obtains a p-value for the implied breakpoint
ci.myts1 <- confint(bp.myts1) #95% CI for the location break date
plot(myts1)
lines(ci.myts1) #This shows the interval around the estimated break date

Using this I can obtain a break date and a 95% CI, which tells me that a break has occurred. However, this break is in the mean since the formula is myts1~1, reflecting a regression on a constant. If I understand this correctly, the residuals are the demeaned values of myts1 and therefore I am looking at a change in the mean. The plot visualizes the data with the breakdate and a confidence interval.

Questions

Q0: Before starting this analysis I was wondering if I should be concerned with how these prediction errors are distributed and correct for certain characteristics? It seems a rather stable process apart from the break occurring and some outliers.

Q1: How can I calculate a change in variance? I can imagine a change in variance could also occur at a different point in time than the mean? Is it correct to say a break in the variance is also a break in the mean, but then a break in the mean of the squared demeaned series? Not much to find about this.

Q2: Given I have now obtained sufficient evidence of a break in mean and variance, how can I quantify this change? e.g. the variance has shifted from X to Y after the break date? Is it as simple as splitting the the time-series along the break date and summarizing statistics about both parts?

Q3: If I rerun the break analysis for other time intervals, how do I compare how the change in mean and variance evolves for the different prediction horizons. Is this yet again a simply summary of the statistics or is there a test that assesses how different the errors are?

addition Q3:##

In creating these time series, prediction errors up to 10 hours before the predicted event occurs are considered.

Taking one day as an example: predictions are seperated into 1 hour bins (creates 10 bins), then within each bin, all predictions are summarized into a weighted average value(weighed based on a different variable). This means that for each day there is one weighted score per bin (total of 10).

Translating this to the time series object that I provided in this post (myts1, covering the last hour) yields the following: A time series in which each point corresponds to the weighted average value for that day in the given time interval. Essentially each bin contains 975 separate days with an average weighted value for each (purely historical).

My thoughts on this part: I added an image containing 9 bins out of 10, which clearly shows that the break becomes less noticeable further back in time. Given these 10 time series, I rerun the "Score-CUSUM" (mean/variance) test for each. From there can be determined at which hour the effect of this system becomes "noticeable" (as in absolute change in mean/variance) and usable from an operational point of view.

Q3.1 Does it make sense to analyze the time series in this way? I assume it does not matter that I re-run the SCORE-CUSUM test 10 times?
Q3.1 How do I deal with a 95% CI that spans 6 months when segmenting the break? (found in bins 4hrs out)
Q3.2 Should I be concerned in comparing the different models (errors) across these 10 time intervals?

I hope my explanation sufficient, can provide more information if necessary.

EDIT: I have added a csv file (seperated by ;) in columnar format, this also includes the number of events that occurred each day, however, there seems to be no correlation when plotted. Link: https://www.dropbox.com/s/5pilmn43bps9ss4/Data.csv?dl=0

EDIT2: Should add that the actual implementation occured around timepoint 2018 day 136 in the timeseries.

EDIT3: Added the second prediction interval of hour 1 to 2 as a TS object in R on pastebin: https://pastebin.com/50sb4RtP (limitations in characters of main post)

Answer 1

Questions

Q0: The time series looks rather right-skewed and the level shift is accompanied by a scale shift. Hence, I would analyze the time series in logs rather than levels, i.e., with multiplicative rather than additive errors. In logs, it seems that an AR(1) model works quite well in each segment. See e.g. acf() and pacf() before and after the break.

pacf(log(window(myts1, end = c(2018, 136))))
pacf(log(window(myts1, start = c(2018, 137))))

Q1: For a time series without breaks in the mean, you can simply use the squared (or absolute) residuals and run a test for level shifts again. Alternatively, you can run tests and breakpoint estimation based on a maximum likelihood model where the error variance is another model parameter in addition to the regression coefficients. This is Zeileis et al. (2010, doi:10.1016/j.csda.2009.12.005). The corresponding score-based CUSUM tests are available in strucchange as well but the breakpoint estimation is in fxregime. Finally, in the absence of regressors when looking only for changes in mean and variance the changepoint R package also provides dedicated functions.

Having said that, it seems that a least-squares approach (treating the variance as a nuisance parameter) is sufficient for the time series you posted. See below.

Q2: Yes. I would simply fit separate models to each segment and analyze these "as usual" Bai & Perron (2003, Journal of Applied Econometrics) also argue that this is justified asymptotically due to the faster convergence of the breakpoint estimates (with rate $n$ rather than $\sqrt{n}$).

Q3: I'm not fully sure what you are looking for here. If you want to run the tests sequentially to monitor incoming data, then you should adopt a formal monitoring approach. This is also discussed in Zeileis et al. (2010).

Analysis code snippets:

Combine log series with its lags for subsequent regression.

d <- ts.intersect(y = log(myts1), y1 = lag(log(myts1), -1))

Testing with supF and score-based CUSUM tests:

fs <- Fstats(y ~ y1, data = d)
plot(fs)
lines(breakpoints(fs))

sc <- efp(y ~ y1, data = d, type = "Score-CUSUM")
plot(sc, functional = NULL)

This highlights that both intercept and autocorrelation coefficient change significantly at the time point visible in the original time series. There is also some fluctuation in the variance but this is not significant at 5% level.

A BIC-based dating also clearly finds this one breakpoint:

bp <- breakpoints(y ~ y1, data = d)
coef(bp)
##                       (Intercept)        y1
## 2016(123) - 2018(136)    3.926381 0.3858473
## 2018(137) - 2019(1)      3.778685 0.2845176

Clearly, the mean drops but also the autocorrelation slightly. The fitted model in logs is then:

plot(log(myts1), col = "lightgray", lwd = 2)
lines(fitted(bp))
lines(confint(bp))

Re-fitting the model to each segments can then be done via:

summary(lm(y ~ y1, data = window(d, end = c(2018, 136))))
## Call:
## lm(formula = y ~ y1, data = window(d, end = c(2018, 136)))
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.73569 -0.18457 -0.04354  0.12042  1.89052 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  3.92638    0.21656   18.13   <2e-16 ***
## y1           0.38585    0.03383   11.40   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.2999 on 742 degrees of freedom
## Multiple R-squared:  0.1491, Adjusted R-squared:  0.148 
## F-statistic: 130.1 on 1 and 742 DF,  p-value: < 2.2e-16

 

summary(lm(y ~ y1, data = window(d, start = c(2018, 137))))
## Call:
## lm(formula = y ~ y1, data = window(d, start = c(2018, 137)))
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.43663 -0.13953 -0.03408  0.09028  0.99777 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  3.61558    0.33468   10.80  < 2e-16 ***
## y1           0.31567    0.06327    4.99  1.2e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.2195 on 227 degrees of freedom
## Multiple R-squared:  0.09883,    Adjusted R-squared:  0.09486 
## F-statistic:  24.9 on 1 and 227 DF,  p-value: 1.204e-06

Answer 2

Not wanting to add too much information in the original post, the reply here is in response to @Achim Zeleis following this part:

"The corresponding score-based CUSUM tests are available in strucchange as well but the breakpoint estimation is in fxregime"

And question 3, which was phrased poorly (now updated in the original post):

"Re: Q3. This sounds indeed like you want to have a "monitoring" or "sequential testing" procedure for which various tools have been proposed under different labels in different communities. Statistical process control or quality control might be another relevant label."

Having read the vignette of fxregime and strucchange a breakdate estimation is obtained.
https://cran.r-project.org/web/packages/fxregime/vignettes/CNY.pdf
https://cran.r-project.org/web/packages/fxregime/fxregime.pdf

The questions associated with this part are as follows:
1. How to translate the one break date estimation found by fxregime to both the changes in intercept and auto correlation?
2. Is the logic/method I used in obtaining this break estimation using fxregime correct?
3. Should I even expect two break dates, or do both changes in intercept and auto correlation occur at the same date? (e.g. what if variance would change at a different break date, do I then get two - three different break dates?
4. Question 3 of the original post applies (updated)
5. Should I be concerned with seasonal effects as suggested by @Irish Stat (deleted answer)? I assume only when I want to model this afterwards, and not during the break testing?

Analysis code snippets for myts1:

Combine log series with its lags for subsequent regression.
d <- ts.intersect(y = log(myts1), y1 = lag(log(myts1), -1))

Taking the same "Score-CUSUM" test:

sc <- efp(y ~ y1, data = d, type = "Score-CUSUM")
plot(sc, functional = NULL)

Break date estimation using fxregime:
1. LWZ and Negative Log−Likelihood plot shows optimal number of breaks of 1
2. Breakdate with a confidence interval indicates a break at observation 744

bd <- fxregimes(y~y1, data = d)
plot(bd) #LWZ and Negative Log-Likelihood plot indicating optimal number of breakpoints is 1 (following vignette information)
ci <- confint(bd, level = 0.95)
ci #show confidence interval for break date(s)

##         Confidence intervals for breakpoints
##         of optimal 2-segment partition: 
##
## Call:
## confint.fxregimes(object = bd, level = 0.95)
##
## Breakpoints at observation number:
##  2.5 % breakpoints 97.5 %
## 1   742         744    746
##
## Corresponding to breakdates:
##     2.5 % breakpoints   97.5 %
## 1 2018.363    2018.369 2018.374

Then with coef I can obtain the coefficients from each segment.

coef(bd)
## 
##                                       (Intercept) y1     (Variance)
## 2016.33334081892--2018.36895725728    3.926381 0.3858473 0.08969063
## 2018.37169698331--2018.99909424358    3.778685 0.2845176 0.04813337

From here I would say that also the variance has dropped by quite a bit, but unsure how to interpret this correctly given a single break date estimation and non significance in the Score-CUSUM test?

Part 2, related to question 3 in the OP

Now as mentioned in Q3 of the (updated) original post, there are multiple time series, the one below is for the prediction of interval 1-2hrs for 975 consecutive days, with each day having one weighted average score.

Analysis code snippets for myts2:
Regarding Q0: re-evaluation of the time series. Referring the second image in the original post, the right skew is still somewhat apparent and looking at the acf() and pacf() before and after the break still indicates that an AR(1) model would work quite well (I think, similar graphs).

pacf(log(window(myts2, end = c(2018, 136))))
pacf(log(window(myts2, start = c(2018, 137))))

Again combine log series with its lags for subsequent regression.
e <- ts.intersect(y = log(myts2), y1 = lag(log(myts2), -1))

"Score-CUSUM" test:

sc2 <- efp(y ~ y1, data = e, type = "Score-CUSUM")
plot(sc2, functional = NULL)

Similar to the first time series, the intercept and auto correlation coefficient change significantly at the time point visible in the original time series. However, this time there is also a fluctuation visible in the variance that is significant at the 5% level, not directly matching the time point of intercept and auto correlation.

Break date estimation using fxregime:
1. LWZ and Negative Log−Likelihood plot shows optimal number of breaks of 1 due to the sharp decline of LWZ and the kink in NLL at after breakpoint 1.
2. Breakdate with a confidence interval indicates a break at observation 736

d <- fxregimes(y~y1, data = d)
plot(bd) #LWZ and Negative Log-Likelihood plot indicating optimal number of breakpoints is 1 (following vignette information)
ci <- confint(bd, level = 0.95)
ci #show confidence interval for break date(s)
# Confidence intervals for breakpoints  

# of optimal 2-segment partition: 
#   
#   Call:
#   confint.fxregimes(object = bd1, level = 0.95)
# 
# Breakpoints at observation number:
#   2.5 % breakpoints 97.5 %
#   1   730         736    750
# 
# Corresponding to breakdates:
#   2.5 % breakpoints   97.5 %
#   1 2018.331    2018.347 2018.385
# > breakdates(ci)
# 2.5 % breakpoints   97.5 %
#   1 2018.331    2018.347 2018.385

Then with coef I can obtain the coefficients from each segment.

coef(bd1)
#                                       (Intercept) y1     (Variance)
# 2016.33334081892--2018.34703944906    3.853897 0.3985997 0.07925990
# 2018.34977917509--2018.99909424358    3.106076 0.4773263 0.04625951

To evaluate this part for myts2 (1-2hr prediction interval), the variance has dropped by quite a bit but the change is less compared to myts1. Moreover, there is a change noticeable in the coefficients of the intercept and the auto correlation.

Also here the question is how should this be interpreted? How does this single break date estimation reflect the breaks that are visually seen in the Score-CUSUM test?

*also saw that the refit function will fit segmented regressions from the fxregimes function, which can be used to compare as mentioned last by @Achim Zeileis.
Is it then possible to compare models (Q3) across the time series (myts1-10)? I assume only when they are sharing the same scale as one cannot compare a model that has a log vs one that does not.

Answer 3

If anybody cares the use of R, a selection of packages available are summarized in the CRAN Task View on Time Series (https://cran.r-project.org/web/views/TimeSeries.html). Here is a relevant excerpt:

Change point detection is provided in strucchange and strucchangeRcpp (using linear regression models) and in trend (using nonparametric tests). The changepoint package provides many popular changepoint methods, and ecp does nonparametric changepoint detection for univariate and multivariate series. changepoint.np implements the nonparametric PELT algorithm, changepoint.mv detects changepoints in multivariate time series, while changepoint.geo implements the high-dimensional changepoint detection method GeomCP. Factor-augmented VAR (FAVAR) models are estimated by a Bayesian method with FAVAR. InspectChangepoint uses sparse projection to estimate changepoints in high-dimensional time series. Rbeast provides Bayesian change-point detection and time series decomposition. breakfast includes methods for fast multiple change-point detection and estimation.