2
$\begingroup$

I am trying to derive a set of equations to model a real-world situation: the analysis of expenditure over time, and I am hitting on a number of barriers (all due to my lack of expertise). Once I have a reasonable methodology, I am likely to be using R, and I wish to derive a number of parameters from this data, if at all possible. I also need to validate whether my theoretical model is accurate, or requires further revision.

Objectives

  1. Determine how much will I have spent in 6 month's time
  2. Estimate the size of the population (which I can derive from the slope of the curve)
  3. State the current growth rate or contraction (and project this figure if possible)

I am approaching this task from two sides: (a) can I develop a theoretical model to fit observed data to; and/or (b) do I derive an equation which describes real-world data? This post mainly focusses on the second aspect.

Part of the trigger for this question is that my data is now showing a decrease in expenditure over time (which my theoretical model does not yet account for). If I plot the running total of expenditure over time for four separate services, and then try to fit my curve to it (previously assumed to be quadratic), I get these results:

Cumulative Spend

What these plots show is that over the past 180days, there has been no expenditure at all for the service in red (Line C), and all the other services are also in various stages of termination.

My initial assumption of trying to fit data to a quadratic curve might be in is an error, as demonstrated in the first image. This is because the slope of the line can NEVER be negative. Money only flows "one way". My original model only considered initial growth having new signups at regular intervals, and not any subsequent decline.

Whilst looking at another dataset, I clearly have an asymmetric sigmoid. With this dataset, I only have the final phase (i.e. the data for the initial growth and maintenance phases are missing). In any observed dataset, I've concluded that I will not actually know which section(s) of the sigmoid function to fit to.

Based on the above, is the best approach to try and develop and refine a theoretical model; or to find a model which best describes observational data? Also, how can I fit to a section of a sigmoid function, since I don't actually know which phase my data is likely to cover?

For further background on how I am trying to approach the theoretical modelling, I've posted a separate question at https://math.stackexchange.com/questions/4995809/derivation-of-cost-projection-formula. Crucially, I have not yet made any allowance for the situation which I am seeing in this case, ie. that the rate of expenditure is falling over time.

If necessary, I can post raw data, but there is rather a lot of it.

Improve this question
$\endgroup$
11
  • $\begingroup$ "My initial assumption of trying to fit data to a quadratic curve might be in error, as demonstrated in the first image. This is because the slope of the line can NEVER be negative." I agree. So, what made you choose a quadratic relationship as your model? I don't understand the derivation in your other question at all. Why does integrating the curve for one person give the cost for all persons? $\endgroup$
    – Roland
    Commented Nov 11 at 6:04
  • $\begingroup$ Expenditure for a single person is (generally) described by y=mx+c - and therefore adding people at regular intervals gives a quadratic. Generally, this simple approach works, except in the situation described in this question. Regarding the derivation of the theoretical curve, I confess that I am really bad at calculus. My approach thus far has been to break the problem into little chunks, solve that, and then to try and refine things when observed does not match theoretical. At this point, I have no model to fit where people are leaving the service early. $\endgroup$
    – Phil
    Commented Nov 11 at 11:25
  • $\begingroup$ I don't believe that is correct. You are neglecting the important fact that the cost curve of a single person is a segmented relationship (with different durations for the linear increase) and not simply linear. Also, the very strong assumption of people being added at regular intervals won't apply for real-world data. $\endgroup$
    – Roland
    Commented Nov 11 at 11:39
  • $\begingroup$ I absolutely agree. This was my starting point, which needs altering. (although that process is proving VERY challenging -- limited by my poor abilities with calculus. Work ongoing... $\endgroup$
    – Phil
    Commented Nov 11 at 16:11
  • 2
    $\begingroup$ I find it interesting to try to model these kind of things. But, as a statistical question it is too much confusing what is going on and what is being asked. Even when we leave out the statistical details and ignore the aspect of variations and errors on the data, then I still don't understand what those curves represent, how they are being created, and why they are being analyzed. $\endgroup$
    – Sextus Empiricus
    Commented Nov 13 at 14:38

1 Answer 1

Reset to default
0
$\begingroup$
  1. Determine how much will I have spent in 6 month's time

You could fit many curves to your plot, but it is not strong for extrapolation. It is better to have a model with a sound theoretical basis whose fitting to some initial time is just to estimate unknown parameters which can be assumed to remain in time. See for example: Why would you perform transformations over polynomial regression? and Interpreting logistic modelling and linear modelling results for the same formula.

It seems like you have an underlying variable, 'the daily increase of costs' which is driven by the number of services. It would be good to focus on that. One helpful analysis might be to look at the derivative of the curve (you can have a smoothened derivative by making a linear fit in a moving window with a Savitzky-Golay filter).

The nature of these 'services' are not exactly clear. There seems to be in general a decline, but it is also not continuous. For example, around the point -100 on the x-axis you have a jump/step in several of the curves.

You could extrapolate this decline in the services, but you need a model for that. How well do the past services predict the future? What do you known about that? You can extrapolate the current slope a bit and include some error computations according to observed statistical deviations based on a linearized fit on the recent curve slopes*, but that doesn't take into account potential changes that are not included in the model.

*Such estimates of statistical variations are not so easy. It is not like a least squares fit and use the mean squared residuals to estimate the variance of an error. The data has autocorrelation because a service is extended in time. The variability seems to be in the probability that a service stops (you could include the condition 'after $k$ days', or 'after $k$ costs', if you know the services individually), and the probability that you get a new service.

A Bayesian model that models underlying latent variables might work well here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.