I have a business in which I contract with my clients to perform work for them for an indefinite amount of time. To forecast my profitability, I need to forecast when my existing contracts will end. I have historical data from which I made a simple function to estimate the likelihood that a contract ends as a function of the age of the contract (how many weeks the contract has been active). This runoff function is a simple table

Age in weeks % of remaining contracts that end (Runoff %)
1 9.26%
2 7.34%
3 5.23%
... ...
208 1.25%

I then take each contract and look up its expected Runoff % for the next week and sum those %'s to get the total number of contracts expected to end. I continue this process for each week in the forecast (increasing each contracts age by 1 as the weeks move forward).

To test whether this is a good estimator, I have looked at forecasting just 1 week into the future. Thus I estimate next week's number of contracts that will end and compare that to the actual number of contracts that ended.

Week Estimate Actual
Jan-4 105 97
Jan-11 106 111
Jan-18 106 107
... ... ...
Dec-27 146 152

What statistical test should I run to determine whether the estimator is a good one and at what confidence level?

I looked at chi-square tests, but they look to apply to categorical data and so does not appear to apply here.

  • 1
    $\begingroup$ It all depends on what "good" means in your application. Why would you need a test? Don't you just want to measure how accurate the estimates are? $\endgroup$ – whuber Apr 30 at 14:25
  • 2
    $\begingroup$ There are various tests for comparing different forecasts for the same actuals, like the Diebold-Mariano test, so you could test whether your forecast is better than some benchmark, like a naive historical average. But as @whuber writes, you will usually just calculate the error of your forecasts and decide whether they are good enough for any decisions you want to base on them. $\endgroup$ – Stephan Kolassa Apr 30 at 14:28
  • $\begingroup$ My sense of "good" is meant to determine whether I need to search for a new estimator. When I looked at the chi-square test, I wanted to be able to say the estimator results and the actual results are likely drawn from the same distribution at a 95% confidence interval. I ran a linear regression with the actuals as the dependent variable and the estimate as the independent. R-squared was 99% and coefficient was 0.998. That seems very good to me, but I would like to make statement regarding confidence intervals if possible. $\endgroup$ – Andrew Apr 30 at 14:55
  • $\begingroup$ An additional consideration for the statistics is I am concerned the distribution changes through time - COVID could certainly have changed client behaviour. $\endgroup$ – Andrew Apr 30 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.