2
$\begingroup$

I am observing a variable (three samples posted below), and I am interested in its cumulative total by the end of the experiment. Here are a few experiments (different color = different experiment) with the plots of the cumulative variable (not the original one observed):

enter image description here

All the plots seem to follow the same skewed-sigmoid pattern, which becomes clearer when I plot the normalized values of the plots above:

enter image description here

Now, I want to model this behavior with the goal of predicting the final cumulative total (where the curve will asymptote) given only the partial information of observing the experiment for, say, the first half of its total duration.

----- Attempt at solution ------

My first idea was to use an AR regression and treat this as a time series problem, however if I want to predict the cumulative total at the end of the experiment using only the information in the first half of the experiment, I simply don't have enough data to train the model to predict that far into the future.

Thus, I would like to follow a parametric approach and use the observation that most of the curves are skewed sigmoids. I can try to fit a generalized sigmoid curve to the data, like the CDF shown here. However, I don't know of any ways to fit a curve knowing only the 'beginning' or 'left-hand side' of it, instead of a more spread out sample.

I hope this makes sense, I will try to clarify if questions come in.

SAMPLE OF UNACCUMULATED DATA:

---------- FIRST SAMPLE ---------

0.00000000e+00
5.37759693e-05
1.25477262e-04
1.43402585e-04
2.15103877e-04
6.27386309e-04
6.45311632e-04
5.73610339e-04
4.66058401e-04
5.19834370e-04
9.67967448e-04
9.85892771e-04
9.32116801e-04
9.50042125e-04
1.00381809e-03
8.78340832e-04
7.88714217e-04
9.14191478e-04
1.36232456e-03
1.79253231e-03
8.96266155e-04
1.34439923e-03
1.05759406e-03
1.00381809e-03
9.50042125e-04
1.29062326e-03
1.20099665e-03
1.16514600e-03
1.02174342e-03
8.78340832e-04
1.05759406e-03
1.12929536e-03
1.12929536e-03
1.46987649e-03
9.85892771e-04
6.63236955e-04
7.17012924e-04
1.23684729e-03
1.32647391e-03
8.60415509e-04
1.11137003e-03
1.70290569e-03
1.39817520e-03
4.66058401e-04
1.25477262e-04
3.22655816e-04
1.79253231e-04
2.50954523e-04
1.43402585e-04
1.25477262e-04
2.33029200e-04
2.33029200e-04
3.58506462e-05
5.37759693e-05
1.07551939e-04
3.58506462e-05
7.17012924e-05

--------- SECOND SAMPLE --------

0.00000000e+00
1.48295345e-05
4.44886035e-05
8.89772070e-05
1.03806742e-04
1.92783949e-04
3.26249759e-04
3.70738363e-04
2.52102087e-04
2.52102087e-04
8.45283467e-04
8.74942536e-04
9.04601605e-04
1.74988507e-03
2.59516854e-03
2.34306645e-03
8.15624398e-04
5.19033708e-04
5.93181380e-04
6.22840449e-04
9.34260674e-04
7.71135794e-04
8.89772070e-04
7.56306260e-04
9.04601605e-04
7.56306260e-04
9.93578812e-04
9.04601605e-04
9.78749277e-04
7.41476725e-04
4.89374639e-04
8.74942536e-04
5.78351846e-04
8.60113001e-04
7.85965329e-04
7.71135794e-04
8.74942536e-04
5.19033708e-04
4.30056501e-04
3.55908828e-04
2.52102087e-04
4.00397432e-04
3.55908828e-04
2.37272552e-04
3.26249759e-04
2.96590690e-04
3.41079294e-04
1.18636276e-04
2.66931621e-04
3.41079294e-04
1.92783949e-04
1.48295345e-05
1.48295345e-05

------ THIRD SAMPLE ------

0.00000000e+00
0.00000000e+00
1.58587514e-04
3.17175028e-04
6.34350056e-04
5.28625046e-04
4.22900037e-04
1.48015013e-03
1.63873764e-03
1.00438759e-03
1.58587514e-03
2.32595020e-03
3.33033779e-03
2.69598774e-03
1.00438759e-03
5.28625046e-04
7.92937569e-04
3.17175028e-04
7.92937569e-04
7.40075065e-04
1.05725009e-03
5.81487551e-04
7.40075065e-04
7.92937569e-04
5.28625046e-04
5.28625046e-04
6.34350056e-04
3.70037532e-04
5.81487551e-04
6.34350056e-04
3.70037532e-04
3.17175028e-04
5.28625046e-04
3.17175028e-04
1.05725009e-04
1.58587514e-04
2.64312523e-04
2.11450019e-04
1.05725009e-04
5.28625046e-05
1.05725009e-04
1.58587514e-04
0.00000000e+00
5.28625046e-05
5.28625046e-05
$\endgroup$
2
  • $\begingroup$ These do not appear to describe "distributions" in the usual sense. It also looks like you are not "observing" a cumulative variable, but perhaps are observing the variable and accumulating it for plotting purposes. These distinctions--especially the latter--are important for understanding and analyzing the situation. Could you clarify what you actually are observing? $\endgroup$ – whuber Oct 2 '19 at 20:02
  • $\begingroup$ @whuber Indeed, I am observing the unaccumulated variable, which I have posted in my three samples above, and accumulate it afterwards for analysis. I have so far been working off of this paper: jstor.org/stable/2634628?seq=1#page_scan_tab_contents Right, I shouldn't use the word distribution. I'll try to edit. $\endgroup$ – Mike Oct 2 '19 at 20:15
2
$\begingroup$

I would not model the cumulative BUT rather the observed data .. the reason being is that the cumulative is TOO auto-correlated due to the successive sums. Forecasts can then be made at the observed level and converted to cumulative forecasts as you wish with probability densities around each future period. Post one of your example observed series and I will try and help further.

We see this problem very frequently as one would like to know the probability of exceeding a particular value at each point in the forecast horizon OR the probability of making a month-end number when analyzing daily data.

You say "partial information of observing the experiment for, say, the first half of its total duration" should be used " . I say the more observations you have the more precise will be your stated probabilities of when you break through some user-specified limit.

EDITED AFTER RECEIPT OF YOUR DATA (SERIES1) I called it A with 57 values and used it as the illustrative example

I used AUTOBOX a piece of software that I have helped to develop and share here the approach, the references and some output yielding a forecast of the OBSERVED DATA from time point 57 which can then be iteratively added to the summed values from period 1-57 to get a distribution of expectations for period 58 .

What is done here can be done by others using other software and a bit of creative coding .

Observed data enter image description here with Actual/Fit and Forecast using a useful ARIMA model with identified determinstic structure (visually obvious ) enter image description here

The model developed by identifying the arima structure and the deterministic structure follows the Box-Jenkins paradigm https://autobox.com/pdfs/ARIMA%20FLOW%20CHART.pdf improved by TSAY http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html is here . It has 2 autoregressive coefficients , 2 time trend indicators , 2 intercept changes , 2 unusual values and an identified error variance changepoint seen here enter image description here at period 42.

enter image description here and here enter image description here

The model residuals are clean enter image description here with an ACF here enter image description here

Thenter image description heree forecasts are here and hereenter image description here

Now to the sweet spot ... We have a set of model residuals which can be used as the basis for a simulation for each of the 36 periods. I show here the simulation for 1 period out enter image description here and for generalization 7 periods out enter image description here

Now if you wish to specify a critical value for the cumulant simply add the first period out simulations to the current cumulant and for any critical value simply reference the cumulative density .

Hope this helps you and others deal with how to "predict the cumulative total " and to estimate probabilities of exceeding user-specified break through points.

$\endgroup$
10
  • $\begingroup$ I've posted three samples of the unaccumulated variable. The reason I went for the accumulated is because it seems to be more stable, the unaccumulated is a bit sparse and harder to conjecture a pattern from. A problem statement where we predict the probability of 'breaking through' a given value (i.e. asymptote for the accumulated variable) would be very useful. Can you point to any sources that expound on this topic? Thanks! $\endgroup$ – Mike Oct 2 '19 at 17:26
  • $\begingroup$ The cumulant analysis is faulty/silly because by the mere act of summing you are filtering/transforming/smoothing the data which then needs to be unravelled by differencing and/or some arima structure The concept of "Fool's Gold" comes to mind. In terms of references all I can say is KISS and like the the Doctor "Do no harm !" . Please repost your 3 examples in a columnar fashion as that is easier for me. What you want in terms of breaking through is quite easy to obtain with forecast confidence limit distributions. $\endgroup$ – IrishStat Oct 2 '19 at 17:33
  • $\begingroup$ never mind your data came in as a row ....it is ok $\endgroup$ – IrishStat Oct 2 '19 at 17:47
  • $\begingroup$ No problem, I just changed them to columnar too $\endgroup$ – Mike Oct 2 '19 at 17:49
  • 1
    $\begingroup$ I have reviewed the article and I would actually have to see what you get when you follow his suggestion . If you wish to offline contact me please use my contact info and I will try and help you better understand. Thanks for your question and nice words ! You can also upvote my answer. $\endgroup$ – IrishStat Oct 3 '19 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.