8
$\begingroup$

Background

So first some background to gauge the level of understanding I might have. Currently completing MSc thesis, statistics has been a negligible part of this although I do have a basic understanding. My current question makes me doubt what I can/should do in practice, reading more and more online and in literature seems to be counterproductive.

What am I trying to achieve?

So for my thesis I joined a company and the general question I am trying to answer there is essentially how a predictive process is affected by the implementation of certain system (which affects the data used for the predictive process).

The desired outcome in this is an understanding of:

  1. Is there a noticeable change? (e.g. statistical proof)
  2. How large is the change? (in mean and variance)
  3. What factors are important in this predictive process (Also how the influence of factors changes from before > after the break)

To answer 1 and 2 I obtained historical data in the form of a time-series object (and more but irrelevant at this stage). The software I use is R.

Data

The data encompasses a weighted score for each day (2.5yrs), indicating how bad the predictive process performed (deviation from the actual event). This one time-series object contains the weighted score for the predictions that occurred from one hour before up until the actual occurrence of the event (1hr interval) for these 2.5 years (so each day has one weighted score for this interval). Likewise, there are multiple time series constructed for other intervals (e.g. 1-2, 2-3 hrs etc.)

myts1 <- structure(c(412.028462047, 468.938224875, 372.353242472, 662.26844965, 
                 526.872020535, 396.434818388, 515.597528222, 536.940884418, 642.878650146, 
                 458.935314286, 544.096691918, 544.378838523, 486.854043968, 478.952935122, 
                 533.171083451, 507.543369365, 475.992539251, 411.626822157, 574.256785085, 
                 489.424743512, 558.03917366, 488.892234577, 1081.570101272, 488.410996801, 
                 420.058151274, 548.43547725, 759.563191992, 699.857042552, 505.546581256, 
                 2399.735167563, 959.058553387, 565.776425823, 794.327364085, 
                 1060.096712241, 636.011672603, 592.842508666, 643.576323635, 
                 639.649884944, 420.788373053, 506.948276856, 503.484363746, 466.642585817, 
                 554.521681602, 578.44355769, 589.29487224, 636.837396631, 647.548662447, 
                 740.222655163, 391.545826142, 537.551842222, 908.940523615, 590.446686171, 
                 543.002925217, 1406.486794264, 1007.596435757, 617.098818856, 
                 633.848676718, 576.040175894, 881.49475483, 687.276105325, 628.977801859, 
                 1398.136047241, 749.644445942, 639.958039461, 649.265606673, 
                 645.57852203, 577.862446744, 663.218073256, 593.034544803, 672.096591437, 
                 544.776355324, 720.242877214, 824.963939263, 596.581822515, 885.215989867, 
                 693.456405627, 552.170633931, 618.855329732, 1030.291011295, 
                 615.889921256, 799.498196448, 570.398558528, 680.670975027, 563.404802085, 
                 494.790365745, 756.684436338, 523.051238729, 535.502475619, 520.8344231, 
                 623.971011973, 928.274580287, 639.702434094, 583.234364572, 623.144865566, 
                 673.342687695, 567.501447619, 602.473664361, 655.181508321, 593.662768316, 
                 617.830786992, 652.461315007, 496.505155747, 550.24687917, 588.952116381, 
                 456.603281447, 425.963966309, 454.729462342, 487.22846023, 613.269432488, 
                 474.916140657, 505.93051487, 536.401546008, 555.824475073, 509.429036303, 
                 632.232746263, 677.102831732, 506.605957979, 701.99882145, 499.770942819, 
                 555.599224002, 557.634152694, 448.693828549, 661.921921922, 447.00540349, 
                 561.194112634, 590.797954608, 590.739061378, 445.949400588, 725.589882976, 
                 480.650749378, 587.03144903, 483.054524693, 428.813155209, 540.609606719, 
                 495.756149832, 409.713220791, 492.43287131, 618.492643291, 723.203623076, 
                 461.433833742, 420.414959481, 480.501175081, 564.955582744, 453.0704893, 
                 506.711353939, 521.12661934, 487.509966405, 483.442305774, 506.932771141, 
                 442.871555249, 873.285819221, 1201.628963682, 1392.479592817, 
                 693.292446258, 629.477998542, 660.777526646, 414.376675251, 475.517946081, 
                 501.626384564, 470.216781646, 444.195433559, 697.258566625, 546.966755779, 
                 428.945521943, 388.203080434, 579.759476551, 548.433130604, 453.950530959, 
                 460.613845164, 534.329569431, 560.663080722, 660.799405665, 432.3134958, 
                 569.59842379, 518.195281689, 650.007266105, 521.642137647, 442.763872575, 
                 687.470213886, 951.651918891, 589.611971045, 493.203713291, 431.966577408, 
                 616.912296912, 685.80916291, 502.518373775, 595.630289879, 563.104035749, 
                 523.383707347, 532.042896625, 470.949823756, 603.408124923, 615.301428799, 
                 708.26541245, 725.853182875, 705.777543119, 530.351781147, 698.828825921, 
                 462.173187592, 366.411986505, 848.613888761, 502.940599188, 456.044881766, 
                 605.321231272, 629.861109863, 431.130428123, 509.672767868, 457.598828697, 
                 553.932034119, 610.181457495, 581.59017099, 540.788638119, 705.226962669, 
                 610.670142045, 566.392016015, 611.086310256, 603.256299175, 766.372982953, 
                 801.921868916, 761.708239486, 580.712445849, 575.53616943, 540.066255921, 
                 608.133122153, 735.063468208, 637.091441112, 778.874033589, 689.350099602, 
                 1003.219851026, 624.107808848, 635.887051641, 420.915060155, 
                 511.460563095, 817.08209288, 603.089908306, 772.6493477, 797.148459813, 
                 588.255963229, 499.050860875, 502.059987, 565.524637543, 1663.182976069, 
                 2281.49950544, 1442.687607103, 1024.355834401, 899.519857882, 
                 988.585993922, 612.834835776, 641.686600038, 717.951451466, 746.441686309, 
                 1147.770724052, 596.279691286, 932.861076555, 497.228997645, 
                 764.895725484, 659.054003787, 1148.227820587, 1403.462969143, 
                 624.733620842, 803.199038618, 839.637983048, 1278.286165347, 
                 774.363457936, 662.767213211, 627.251799204, 650.180035442, 1296.405174964, 
                 662.928010153, 523.095967567, 620.727894789, 650.876097695, 509.534317267, 
                 479.922326477, 613.743251306, 430.117763379, 1825.108688714, 
                 744.708270099, 455.818978039, 370.908485795, 771.317824437, 688.219350724, 
                 468.16351523, 791.649828808, 666.360829114, 1427.809117119, 2861.163543428, 
                 1090.887950582, 621.942045727, 397.381382335, 397.697308586, 
                 494.441558442, 474.314526966, 888.812606506, 476.031636688, 651.907747324, 
                 389.95997873, 680.776897408, 1499.093314237, 1077.571595752, 
                 765.690897368, 571.545469449, 590.64855754, 492.371592484, 580.811781306, 
                 873.628734717, 602.958435426, 549.877214337, 546.66120979, 394.75285753, 
                 520.238244635, 517.217468365, 903.057976974, 528.477241796, 378.958677302, 
                 491.589659729, 548.665964908, 453.512746452, 481.081050678, 491.499714029, 
                 628.539705456, 672.540312912, 1686.825394554, 1367.577856001, 
                 600.373039737, 417.511405109, 511.75535978, 440.677427555, 493.430816323, 
                 533.025975459, 547.429120615, 432.168874608, 555.098163047, 521.644301834, 
                 667.159371501, 421.591007887, 757.218378664, 615.572602597, 433.961482908, 
                 528.813953729, 633.228715271, 519.648748842, 437.342815473, 551.877832301, 
                 703.377801948, 536.673383258, 658.597165739, 1449.850501569, 
                 615.204142853, 499.197033946, 853.692014263, 490.213941347, 812.68824521, 
                 521.364349414, 818.757704456, 848.59674442, 646.819554339, 471.051626838, 
                 598.326620222, 782.58569568, 754.880939869, 636.572395084, 686.076138643, 
                 530.158582782, 524.696479569, 525.441231521, 593.834663615, 415.830854949, 
                 590.135594493, 591.019407595, 503.321975981, 515.371205208, 494.805384342, 
                 567.397190671, 482.180658052, 724.099533838, 791.107121538, 564.673191002, 
                 572.551388184, 729.46937136, 943.538757014, 519.051645932, 994.190842696, 
                 866.69659257, 610.021553913, 547.791568399, 578.854543644, 684.826681706, 
                 815.179238308, 617.050464226, 623.818649573, 537.163825262, 529.850027242, 
                 926.531531345, 588.578930644, 457.329084489, 380.160216157, 494.287689357, 
                 463.885244047, 451.611520014, 762.508948042, 773.74942889, 1642.691010358, 
                 555.226392541, 659.433830806, 454.348720108, 388.274823265, 650.63824747, 
                 632.327400443, 584.93699748, 484.815917524, 733.153950316, 471.349864174, 
                 418.755413722, 547.060192029, 742.028289483, 521.119798289, 1176.207996336, 
                 524.730544122, 430.009783422, 558.479383664, 574.162550914, 526.08247269, 
                 611.207728202, 551.202548069, 472.046973518, 517.490179087, 556.135143079, 
                 628.084374004, 413.677676623, 439.814082201, 1011.775306843, 
                 684.443831473, 546.421742134, 578.853727684, 517.693483714, 638.112468944, 
                 631.531739664, 501.897019514, 661.11860926, 521.695715961, 474.403897254, 
                 463.294645328, 559.583511974, 531.953658919, 740.412596176, 534.815607516, 
                 462.329096628, 637.941748843, 702.69170843, 471.390065606, 590.458408612, 
                 617.006573387, 565.411288964, 472.986933034, 567.745850996, 596.925622448, 
                 474.068038429, 653.56453828, 612.893376781, 711.545758298, 527.783301631, 
                 478.530081662, 519.751192408, 536.550807025, 443.437342694, 587.403769673, 
                 601.15805729, 556.497167238, 374.228230116, 477.027420471, 494.984999444, 
                 879.314339401, 704.997313272, 626.546803934, 653.296523326, 435.581408863, 
                 633.048339362, 403.889616794, 488.214190958, 575.631003993, 430.984422675, 
                 437.83561603, 522.277281965, 475.602597701, 527.12160277, 944.139469794, 
                 474.50403295, 579.478722386, 459.088134733, 503.246692031, 610.022771263, 
                 446.143895372, 625.022916127, 517.435543013, 891.375454252, 555.864115385, 
                 474.764739145, 921.714956231, 645.896256587, 1536.221634415, 
                 816.575921465, 596.491670621, 503.56011064, 720.743463226, 905.835642175, 
                 1360.481537034, 653.224092421, 633.505228314, 546.064475635, 
                 482.454025258, 962.715357696, 618.202090733, 803.895156435, 668.047995992, 
                 594.566585046, 839.597813143, 457.375793588, 631.863607862, 475.266615122, 
                 664.569635822, 481.886574644, 1614.962054217, 869.212340286, 
                 501.400781534, 478.670649186, 521.824073342, 684.720851031, 597.124676952, 
                 605.903108456, 491.358096619, 430.812042311, 388.350092055, 488.132638097, 
                 413.131448595, 391.891460495, 430.760685279, 731.99097305, 382.200799877, 
                 511.48361093, 560.620999712, 528.369543055, 536.348770159, 721.297750609, 
                 491.321646454, 509.521489714, 561.318889907, 553.24041301, 459.235996646, 
                 354.741174128, 339.775552834, 432.548724483, 438.672630955, 508.177204773, 
                 496.199702536, 643.867549669, 611.460979278, 861.190516859, 662.56052508, 
                 524.398593443, 529.585928069, 607.575374022, 495.001029442, 700.371352785, 
                 794.753142167, 466.792229932, 435.426320832, 450.903747896, 622.562955777, 
                 1562.215153595, 725.069249874, 612.357398912, 418.579228487, 
                 381.667629501, 528.173266471, 687.876352966, 655.845568131, 423.589678964, 
                 612.545707971, 951.362478322, 1800.162370822, 600.672989388, 
                 531.048286916, 527.565406977, 402.380659606, 607.699770367, 1486.296473731, 
                 686.560841226, 4176.136413427, 3086.067140966, 1872.815975088, 
                 771.413460362, 843.791946967, 652.825527602, 642.443948966, 726.208291336, 
                 641.092848676, 488.237988698, 606.154989706, 1426.027951807, 
                 959.347533388, 649.856202928, 527.580884911, 400.545393834, 568.268813107, 
                 631.257023117, 515.755741256, 682.375587555, 583.855170876, 506.146152757, 
                 517.095094378, 563.415777949, 801.015579658, 649.56360904, 732.097267107, 
                 456.626323752, 499.170138889, 549.393587002, 556.589070013, 590.180621262, 
                 667.709332802, 421.738377899, 661.178862228, 570.833727593, 631.139001868, 
                 545.835879493, 559.918523671, 1364.379214546, 985.777069008, 
                 644.949427255, 493.066294248, 476.852498787, 379.716401582, 715.333935018, 
                 459.326945313, 621.665546323, 476.317803131, 519.803138696, 409.241665463, 
                 465.206511176, 594.689036224, 443.841857849, 399.830019307, 570.65982956, 
                 516.562325113, 381.909941529, 532.130831616, 650.329631588, 661.055942562, 
                 1136.942413908, 508.543555485, 976.852889691, 1461.16921717, 
                 646.062436059, 593.093537367, 624.839875084, 453.453385269, 584.633165187, 
                 507.616009915, 516.857276979, 434.651983821, 572.755844368, 454.901132196, 
                 707.698546138, 760.341584614, 449.252091224, 623.217222998, 625.061550699, 
                 2030.045687713, 1582.036383383, 677.325281969, 571.588930686, 
                 493.235172445, 556.291968991, 424.360693057, 436.333980583, 484.105667103, 
                 505.231040152, 378.767240615, 495.943549377, 321.856525703, 363.651848067, 
                 557.201599565, 603.658298878, 558.958198405, 789.717963533, 480.370977054, 
                 509.366153138, 467.526623793, 576.508422894, 661.322171003, 520.804998847, 
                 342.109381368, 473.512224982, 984.139466992, 487.586712759, 605.914245454, 
                 459.190981983, 678.728907858, 342.511103348, 436.746013478, 520.896987467, 
                 818.078350515, 527.494249096, 713.52499017, 610.365469264, 462.965548015, 
                 362.931986459, 810.610193032, 393.455578799, 536.720944152, 551.490260933, 
                 464.369987186, 275.832746918, 513.723009815, 491.945195301, 438.865839297, 
                 257.252871794, 615.513481211, 420.507536576, 392.035094971, 392.963333027, 
                 435.276624468, 253.431425091, 592.873595776, 500.615067792, 503.491101855, 
                 475.352827724, 1135.11762886, 723.666909467, 712.259187274, 559.738346197, 
                 490.958692763, 435.998397207, 729.341315271, 406.369683231, 632.626098862, 
                 565.318329487, 394.031553179, 356.627786519, 374.075606064, 336.505546227, 
                 393.168901965, 480.183256037, 573.840777708, 187.680483645, 170.978544639, 
                 209.134883957, 193.039610198, 224.362544607, 210.946012575, 166.006351727, 
                 201.500604051, 160.008039339, 229.847327915, 193.655724693, 255.575881835, 
                 207.0547762, 186.835573846, 247.776981217, 574.489201771, 602.683083111, 
                 202.592131296, 205.827635803, 279.139480224, 367.898232724, 245.75114848, 
                 304.004111775, 231.706217872, 159.750034155, 207.584399879, 239.854118732, 
                 212.711257914, 181.755353255, 225.98761652, 206.570825973, 193.940523176, 
                 203.851963746, 170.700301226, 226.41517367, 183.87542452, 206.640800387, 
                 218.953856594, 176.127748987, 168.205878516, 201.718224491, 171.421791143, 
                 193.065350731, 235.118975292, 217.002358158, 197.563916607, 219.266588546, 
                 168.044251777, 182.1752633, 172.514392523, 192.250197416, 171.023249603, 
                 151.848998035, 151.678097242, 173.28609604, 194.852548067, 178.742929426, 
                 180.591580951, 189.241987711, 206.960325123, 207.639468123, 307.389246296, 
                 197.203559504, 201.143394951, 313.861245405, 215.87226561, 218.500147257, 
                 201.993836847, 204.461813487, 180.04803872, 305.301279156, 214.880173012, 
                 250.930811019, 284.90370816, 212.518096229, 195.034955147, 190.236951118, 
                 257.088555132, 189.502428671, 239.652450054, 243.53269907, 289.820554649, 
                 249.011846483, 279.393483142, 194.006529415, 214.59905786, 247.746491522, 
                 224.723467084, 191.086994265, 271.654470268, 183.455652269, 327.307813653, 
                 149.721791725, 170.538132352, 192.954323882, 172.459631548, 199.829129152, 
                 163.257916856, 179.33550074, 207.618752518, 191.969484656, 191.506188821, 
                 176.446694609, 194.818608621, 167.891653012, 167.000377869, 201.869893803, 
                 183.895614224, 148.510174938, 157.27033831, 166.573696008, 309.557485177, 
                 595.559287832, 232.318746483, 188.295593138, 157.721093739, 177.753625321, 
                 214.844940799, 226.812056373, 213.910815993, 188.973729833, 243.306529363, 
                 202.733401293, 197.296870477, 182.523989567, 170.428625204, 172.593507241, 
                 151.926342178, 283.415923815, 163.456408737, 172.273699114, 159.382452668, 
                 182.158974043, 156.532047405, 182.768725504, 239.318216814, 151.579013694, 
                 148.216885776, 241.346766195, 257.110058666, 147.931152011, 186.2508302, 
                 197.379372605, 155.69798218, 147.370235658, 180.961415272, 168.372900875, 
                 297.9454882, 177.640856836, 170.535749604, 166.211309731, 178.463064513, 
                 175.035906056, 179.971897406, 162.858094293, 164.056675362, 250.151933362, 
                 195.49943188, 175.281720263, 182.392087534, 144.341091392, 241.466298465, 
                 184.806150673, 167.932448156, 150.621075732, 178.510315062, 209.913152992, 
                 222.798716921, 153.536476868, 190.326281004, 173.708412622, 148.075945402, 
                 149.147879365, 181.5851776, 141.012604897, 157.982751461, 159.364653673, 
                 151.777006477, 191.065323099, 139.076429662, 172.452344359, 182.482693518, 
                 287.424045314, 169.327943406, 298.970852166, 145.238645275, 175.211717467, 
                 165.116950464, 187.006564993, 160.959119092, 198.552333833, 170.406040157, 
                 162.10770001, 146.356131036, 170.336552623, 163.095730104, 155.192077125, 
                 217.182324478, 231.270198833, 256.905250226, 184.59676245, 175.557146633, 
                 164.246298131, 221.695058452, 197.911691457, 188.427830442, 259.361745153, 
                 164.243672823, 190.67188784, 182.331604811, 190.352555581, 248.738493256, 
                 196.854564795, 164.974185334, 332.650385373, 169.081552611, 193.578840033, 
                 192.166911863, 214.174943222, 271.287900593, 224.675083031, 171.950208574, 
                 173.867031268, 139.260432794, 177.012491325, 171.268066406, 132.714578168, 
                 197.224558817, 152.561299656, 143.415562042), .Tsp = c(2016.3306010929, 
                                                                        2018.99909424358, 365), class = "ts")

Process until now

Now I had understood that for question 1 I can apply a test for a structural break to determine if and when the break occurred (with a known break date). For this I use strucchange package in R and utilize the breakpoints function.

However, the CUSUM (for unkown break date) test was also recommended by my supervisor. Unsure what is best here?

EDIT:

I see Andrew's supF test conducts the Chow's test for all possible breaks. Then rejects if the maximum of the F (or Chow) statistics become too large. (Found - perform chow test on time series)

Code to obtain a break date using struccchange

library(strucchange)
test2 <- Fstats(myts1~1) #Gets a sequence of fstatistics for all possible 
# break points within the middle 70% of myts1
myts1.fs <- test2$Fstats #These are the fstats
bp.myts1 <- breakpoints(myts1~1) #Gets the breakpoint based on the F-stats
plot(myts1) #plots the series myts1
lines(bp.myts1) #plots the break date implied by the sup F test
bd.myts1 <- breakdates(bp.myts1) #Obtains the implied break data (2018.35, 
# referring to day 128 (0.35*365 = day number))
sctest(test2) #Obtains a p-value for the implied breakpoint
ci.myts1 <- confint(bp.myts1) #95% CI for the location break date
plot(myts1)
lines(ci.myts1) #This shows the interval around the estimated break date

Using this I can obtain a break date and a 95% CI, which tells me that a break has occurred. However, this break is in the mean since the formula is myts1~1, reflecting a regression on a constant. If I understand this correctly, the residuals are the demeaned values of myts1 and therefore I am looking at a change in the mean. The plot visualizes the data with the breakdate and a confidence interval.

plot

Questions

Q0: Before starting this analysis I was wondering if I should be concerned with how these prediction errors are distributed and correct for certain characteristics? It seems a rather stable process apart from the break occurring and some outliers.

Q1: How can I calculate a change in variance? I can imagine a change in variance could also occur at a different point in time than the mean? Is it correct to say a break in the variance is also a break in the mean, but then a break in the mean of the squared demeaned series? Not much to find about this.

Q2: Given I have now obtained sufficient evidence of a break in mean and variance, how can I quantify this change? e.g. the variance has shifted from X to Y after the break date? Is it as simple as splitting the the time-series along the break date and summarizing statistics about both parts?

Q3: If I rerun the break analysis for other time intervals, how do I compare how the change in mean and variance evolves for the different prediction horizons. Is this yet again a simply summary of the statistics or is there a test that assesses how different the errors are?

addition Q3:##

In creating these time series, prediction errors up to 10 hours before the predicted event occurs are considered.

Taking one day as an example: predictions are seperated into 1 hour bins (creates 10 bins), then within each bin, all predictions are summarized into a weighted average value(weighed based on a different variable). This means that for each day there is one weighted score per bin (total of 10).

Translating this to the time series object that I provided in this post (myts1, covering the last hour) yields the following: A time series in which each point corresponds to the weighted average value for that day in the given time interval. Essentially each bin contains 975 separate days with an average weighted value for each (purely historical).

My thoughts on this part: I added an image containing 9 bins out of 10, which clearly shows that the break becomes less noticeable further back in time. Given these 10 time series, I rerun the "Score-CUSUM" (mean/variance) test for each. From there can be determined at which hour the effect of this system becomes "noticeable" (as in absolute change in mean/variance) and usable from an operational point of view.

enter image description here

Q3.1 Does it make sense to analyze the time series in this way? I assume it does not matter that I re-run the SCORE-CUSUM test 10 times?
Q3.1 How do I deal with a 95% CI that spans 6 months when segmenting the break? (found in bins 4hrs out)
Q3.2 Should I be concerned in comparing the different models (errors) across these 10 time intervals?

I hope my explanation sufficient, can provide more information if necessary.

EDIT: I have added a csv file (seperated by ;) in columnar format, this also includes the number of events that occurred each day, however, there seems to be no correlation when plotted. Link: https://www.dropbox.com/s/5pilmn43bps9ss4/Data.csv?dl=0

EDIT2: Should add that the actual implementation occured around timepoint 2018 day 136 in the timeseries.

EDIT3: Added the second prediction interval of hour 1 to 2 as a TS object in R on pastebin: https://pastebin.com/50sb4RtP (limitations in characters of main post)

$\endgroup$
  • $\begingroup$ Maybe look into garch models which are used (or so I am told) in finance to model (and predict) dynamic variance? Search this site, it is worth a try! $\endgroup$ – kjetil b halvorsen Mar 5 at 20:41
  • $\begingroup$ A structural break in time series can happen a number of ways .. 1. points in time where the model implied intercept changes 2. points in time where the trend changes 3. points in time where one or more of the arma parameters change 4. points in time where the error variance changes deterministically 5. there can be a day-of-the-week, a day-of-the-month,a week of the month, a month-of-the-year, effects around holidays etc that may need to be controlled for. SimpLe chow tests for a mean shift are quite naive. if you post your data in a columnar fashion as a csv file I might be able to help furt $\endgroup$ – IrishStat Mar 7 at 1:32
  • $\begingroup$ @kjetilbhalvorsen I have seen these models be suggested in financial settings, however, I have not been convinced after a read up and discussion with my academic supervisor to this. Thank you for the suggestion! $\endgroup$ – Bas Mar 8 at 8:37
  • $\begingroup$ @IrishStat Was wondering whether there are seasonal effects as well (will look into this today and add more information as I find it). Added a link to a csv file using dropbox in the main post. $\endgroup$ – Bas Mar 8 at 8:37
9
$\begingroup$

Questions

Q0: The time series looks rather right-skewed and the level shift is accompanied by a scale shift. Hence, I would analyze the time series in logs rather than levels, i.e., with multiplicative rather than additive errors. In logs, it seems that an AR(1) model works quite well in each segment. See e.g. acf() and pacf() before and after the break.

pacf(log(window(myts1, end = c(2018, 136))))
pacf(log(window(myts1, start = c(2018, 137))))

Q1: For a time series without breaks in the mean, you can simply use the squared (or absolute) residuals and run a test for level shifts again. Alternatively, you can run tests and breakpoint estimation based on a maximum likelihood model where the error variance is another model parameter in addition to the regression coefficients. This is Zeileis et al. (2010, doi:10.1016/j.csda.2009.12.005). The corresponding score-based CUSUM tests are available in strucchange as well but the breakpoint estimation is in fxregime. Finally, in the absence of regressors when looking only for changes in mean and variance the changepoint R package also provides dedicated functions.

Having said that, it seems that a least-squares approach (treating the variance as a nuisance parameter) is sufficient for the time series you posted. See below.

Q2: Yes. I would simply fit separate models to each segment and analyze these "as usual" Bai & Perron (2003, Journal of Applied Econometrics) also argue that this is justified asymptotically due to the faster convergence of the breakpoint estimates (with rate $n$ rather than $\sqrt{n}$).

Q3: I'm not fully sure what you are looking for here. If you want to run the tests sequentially to monitor incoming data, then you should adopt a formal monitoring approach. This is also discussed in Zeileis et al. (2010).

Analysis code snippets:

Combine log series with its lags for subsequent regression.

d <- ts.intersect(y = log(myts1), y1 = lag(log(myts1), -1))

Testing with supF and score-based CUSUM tests:

fs <- Fstats(y ~ y1, data = d)
plot(fs)
lines(breakpoints(fs))

Fstats

sc <- efp(y ~ y1, data = d, type = "Score-CUSUM")
plot(sc, functional = NULL)

gefp

This highlights that both intercept and autocorrelation coefficient change significantly at the time point visible in the original time series. There is also some fluctuation in the variance but this is not significant at 5% level.

A BIC-based dating also clearly finds this one breakpoint:

bp <- breakpoints(y ~ y1, data = d)
coef(bp)
##                       (Intercept)        y1
## 2016(123) - 2018(136)    3.926381 0.3858473
## 2018(137) - 2019(1)      3.778685 0.2845176

Clearly, the mean drops but also the autocorrelation slightly. The fitted model in logs is then:

plot(log(myts1), col = "lightgray", lwd = 2)
lines(fitted(bp))
lines(confint(bp))

breakpoints

Re-fitting the model to each segments can then be done via:

summary(lm(y ~ y1, data = window(d, end = c(2018, 136))))
## Call:
## lm(formula = y ~ y1, data = window(d, end = c(2018, 136)))
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.73569 -0.18457 -0.04354  0.12042  1.89052 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  3.92638    0.21656   18.13   <2e-16 ***
## y1           0.38585    0.03383   11.40   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.2999 on 742 degrees of freedom
## Multiple R-squared:  0.1491, Adjusted R-squared:  0.148 
## F-statistic: 130.1 on 1 and 742 DF,  p-value: < 2.2e-16

 

summary(lm(y ~ y1, data = window(d, start = c(2018, 137))))
## Call:
## lm(formula = y ~ y1, data = window(d, start = c(2018, 137)))
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.43663 -0.13953 -0.03408  0.09028  0.99777 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  3.61558    0.33468   10.80  < 2e-16 ***
## y1           0.31567    0.06327    4.99  1.2e-06 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.2195 on 227 degrees of freedom
## Multiple R-squared:  0.09883,    Adjusted R-squared:  0.09486 
## F-statistic:  24.9 on 1 and 227 DF,  p-value: 1.204e-06
$\endgroup$
  • $\begingroup$ Thank you for your detailed response. Q1: I understand using the squared residuals, running the test again detects a break in variance, but only without breaks in the mean. I was advised towards a paper that looks at this problem when a break in the mean is present. The paper proposes a special test, however my knowledge in assessing the content is not sufficient: tandfonline.com/doi/abs/10.1198/073500103288619269 (section 3.1) Thus I wonder if this option should be explored, or keep the path of using the score-based CUSUM as you had outlined. Q3: adding clarification in main post $\endgroup$ – Bas Mar 11 at 8:24
  • 1
    $\begingroup$ My impression re Q1 is that most tests will find clear evidence for a break. And given the break you can then characterize the resulting segments concerning their mean, variance, autocorrelation, etc. Re: Q3. This sounds indeed like you want to have a "monitoring" or "sequential testing" procedure for which various tools have been proposed under different labels in different communities. Statistical process control or quality control might be another relevant label. $\endgroup$ – Achim Zeileis Mar 11 at 9:29
  • $\begingroup$ RE Q3: This makes sense and is also highlighted in Zeileis et al. (2010). I will have a look and see how I can apply this, thank you. $\endgroup$ – Bas Mar 11 at 9:39
  • $\begingroup$ re Q3: I may have created some confusion with my edit regarding the data itself. I had mentioned that these aggregated errors are for a given time interval, justifiably you suggested a sequential test to monitor incoming data as it becomes available (reading from Strucchange vignette and Zeileis et al. (2010)). I updated Q3 in the main post, which now (from my understanding) does not work with a sequential test. $\endgroup$ – Bas Mar 11 at 20:13
3
$\begingroup$

Not wanting to add too much information in the original post, the reply here is in response to @Achim Zeleis following this part:

"The corresponding score-based CUSUM tests are available in strucchange as well but the breakpoint estimation is in fxregime"

And question 3, which was phrased poorly (now updated in the original post):

"Re: Q3. This sounds indeed like you want to have a "monitoring" or "sequential testing" procedure for which various tools have been proposed under different labels in different communities. Statistical process control or quality control might be another relevant label."

Having read the vignette of fxregime and strucchange a breakdate estimation is obtained.
https://cran.r-project.org/web/packages/fxregime/vignettes/CNY.pdf
https://cran.r-project.org/web/packages/fxregime/fxregime.pdf

The questions associated with this part are as follows:
1. How to translate the one break date estimation found by fxregime to both the changes in intercept and auto correlation?
2. Is the logic/method I used in obtaining this break estimation using fxregime correct?
3. Should I even expect two break dates, or do both changes in intercept and auto correlation occur at the same date? (e.g. what if variance would change at a different break date, do I then get two - three different break dates?
4. Question 3 of the original post applies (updated)
5. Should I be concerned with seasonal effects as suggested by @Irish Stat (deleted answer)? I assume only when I want to model this afterwards, and not during the break testing?

Analysis code snippets for myts1:

Combine log series with its lags for subsequent regression.
d <- ts.intersect(y = log(myts1), y1 = lag(log(myts1), -1))

Taking the same "Score-CUSUM" test:

sc <- efp(y ~ y1, data = d, type = "Score-CUSUM")
plot(sc, functional = NULL)

Score-based CUSUM test for sc

Break date estimation using fxregime:
1. LWZ and Negative Log−Likelihood plot shows optimal number of breaks of 1
2. Breakdate with a confidence interval indicates a break at observation 744

bd <- fxregimes(y~y1, data = d)
plot(bd) #LWZ and Negative Log-Likelihood plot indicating optimal number of breakpoints is 1 (following vignette information)
ci <- confint(bd, level = 0.95)
ci #show confidence interval for break date(s)

##         Confidence intervals for breakpoints
##         of optimal 2-segment partition: 
##
## Call:
## confint.fxregimes(object = bd, level = 0.95)
##
## Breakpoints at observation number:
##  2.5 % breakpoints 97.5 %
## 1   742         744    746
##
## Corresponding to breakdates:
##     2.5 % breakpoints   97.5 %
## 1 2018.363    2018.369 2018.374

LWZ and Negative Log−Likelihood bd

Then with coef I can obtain the coefficients from each segment.

coef(bd)
## 
##                                       (Intercept) y1     (Variance)
## 2016.33334081892--2018.36895725728    3.926381 0.3858473 0.08969063
## 2018.37169698331--2018.99909424358    3.778685 0.2845176 0.04813337

From here I would say that also the variance has dropped by quite a bit, but unsure how to interpret this correctly given a single break date estimation and non significance in the Score-CUSUM test?

Part 2, related to question 3 in the OP

Now as mentioned in Q3 of the (updated) original post, there are multiple time series, the one below is for the prediction of interval 1-2hrs for 975 consecutive days, with each day having one weighted average score.

Analysis code snippets for myts2:
Regarding Q0: re-evaluation of the time series. Referring the second image in the original post, the right skew is still somewhat apparent and looking at the acf() and pacf() before and after the break still indicates that an AR(1) model would work quite well (I think, similar graphs).

pacf(log(window(myts2, end = c(2018, 136))))
pacf(log(window(myts2, start = c(2018, 137))))

Again combine log series with its lags for subsequent regression.
e <- ts.intersect(y = log(myts2), y1 = lag(log(myts2), -1))

"Score-CUSUM" test:

sc2 <- efp(y ~ y1, data = e, type = "Score-CUSUM")
plot(sc2, functional = NULL)

Score-based CUSUM test for sc2 Similar to the first time series, the intercept and auto correlation coefficient change significantly at the time point visible in the original time series. However, this time there is also a fluctuation visible in the variance that is significant at the 5% level, not directly matching the time point of intercept and auto correlation.

Break date estimation using fxregime:
1. LWZ and Negative Log−Likelihood plot shows optimal number of breaks of 1 due to the sharp decline of LWZ and the kink in NLL at after breakpoint 1.
2. Breakdate with a confidence interval indicates a break at observation 736

d <- fxregimes(y~y1, data = d)
plot(bd) #LWZ and Negative Log-Likelihood plot indicating optimal number of breakpoints is 1 (following vignette information)
ci <- confint(bd, level = 0.95)
ci #show confidence interval for break date(s)
# Confidence intervals for breakpoints  

# of optimal 2-segment partition: 
#   
#   Call:
#   confint.fxregimes(object = bd1, level = 0.95)
# 
# Breakpoints at observation number:
#   2.5 % breakpoints 97.5 %
#   1   730         736    750
# 
# Corresponding to breakdates:
#   2.5 % breakpoints   97.5 %
#   1 2018.331    2018.347 2018.385
# > breakdates(ci)
# 2.5 % breakpoints   97.5 %
#   1 2018.331    2018.347 2018.385

LWZ and Negative Log−Likelihood bd1
Then with coef I can obtain the coefficients from each segment.

coef(bd1)
#                                       (Intercept) y1     (Variance)
# 2016.33334081892--2018.34703944906    3.853897 0.3985997 0.07925990
# 2018.34977917509--2018.99909424358    3.106076 0.4773263 0.04625951

To evaluate this part for myts2 (1-2hr prediction interval), the variance has dropped by quite a bit but the change is less compared to myts1. Moreover, there is a change noticeable in the coefficients of the intercept and the auto correlation.

Also here the question is how should this be interpreted? How does this single break date estimation reflect the breaks that are visually seen in the Score-CUSUM test?

*also saw that the refit function will fit segmented regressions from the fxregimes function, which can be used to compare as mentioned last by @Achim Zeileis.
Is it then possible to compare models (Q3) across the time series (myts1-10)? I assume only when they are sharing the same scale as one cannot compare a model that has a log vs one that does not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.