3
$\begingroup$

One of the things I've struggled with is the issue of effectively summarizing some sort of time series metric (i.e. independent variable) for use in a regression analysis. The most common solution that I've encountered (which may not be the best) is to compute a meaningful average. Although I can get a single number I still have the nagging feeling that it doesn't really capture what I need.

Here's an example graph: enter image description here

Let's assume that there are 3 data points which when plotted over time we can see different types of growth. What I wish to capture/model is that the "flattens" represents the possibility that item sale (or whatever it represents) is eventually going to take a downturn or just stop selling. The "growth" curve is something that won't immediately stop but for the time being (e.g. 1 month) it's a safe bet to assume item popularity.

Here are my questions:

  1. How can/should we capture the "shapes" of such time series based independent variables for regression? Does it even matter or is the average the best we can do?
  2. I thought of computing "slopes" (start, end) but as the graph shows (approximately) that there such a computation could lead to similar values for both flattens and growth curves. Is this even a thing?
  3. What could/should I use?

Note: These shapes are just examples and the actual time series could be rather fluctuating. What I think I'm interested in is a way to capture a notion of a general trend in some way (if at all). I understand trends could be captured via moving averages, but I don't know how to capture "up/down-ness" or summarize it in a meaningful way.

I'm hoping this doesn't become a two-step problem where we first train a model on "learning" these curves and what they represent and then feeding the output into the regression. That would be too complicated for now but I don't even know if that's a valid solution or should I just stick with averages.

$\endgroup$
3
  • $\begingroup$ The joint distribution over the variables in a stochastic process gives, in a sense, its shape. Further mathematical treatment would be required to describe the "linear", "flattening", and "growth" shapes you illustrated in terms of distribution shape. $\endgroup$
    – Galen
    Apr 25, 2023 at 19:43
  • $\begingroup$ You could also try bringing total curvature into a statistical context, or at least a discrete analog of it. Relatedly, you might want to check out discrete differential geometry and see if there is a statistical treatment of it. $\endgroup$
    – Galen
    Apr 25, 2023 at 19:48
  • $\begingroup$ @Galen - your first comment is too loaded for me to parse (not a native statistical speaker :D). I'll check out the link...my goal is to keep things as simple as necessary and still capture the essence of things. $\endgroup$
    – PhD
    Apr 25, 2023 at 20:21

1 Answer 1

2
$\begingroup$

One possible approach that I've been investigating is the use of the second derivative (double differencing) with averaging and/or summing the values (more on that in a bit).

Here's an example in Python:

import numpy as np
d = np.array([0,7.5,9,9,9,8,8,8,8,8,8,8]) #flattens curve
avg = np.average(np.diff(np.diff(d)))     # = -0.75
sum = np.sum(np.diff(np.diff(d)))         # = -7.5

The average provides better "trend" of the second derivative IMHO. The summation seems better at "locating" the approximate local maxima/minima "point".

Here's how to interpret the values:

avg = 0 -> flat trend
avg < 0 -> decreasing trend
avg > 0 -> increasing trend

We could make this a little more tolerant by assuming flatness is between $\pm 1$ if necessary. For all different time series patterns (more than about a 100 in my data set) this seems to provide a decent solution to address the need that is at the right level of simplicity.

For now, this is what I will be going with unless someone provides a better answer. I may be inclined to include both the "overall average" of the entire time series independent variable plus the average of the second derivative. I will surely do a correlation test to ascertain if I should only keep one of them, but I have a feeling that both could be valuable (though I'm not sure if this commonly practiced).

$\endgroup$
7
  • $\begingroup$ Is the distinction clear in your approach between monotonicity and curvature? You seem to be discussing both but they are not logically dependent. $\endgroup$
    – Galen
    Apr 26, 2023 at 23:03
  • $\begingroup$ I'm not really sure if I can give a clear cut answer to that since I don't understand what it is you're getting it (I have limited statistical brain power :(). Perhaps if you can provide an example to help me understand? $\endgroup$
    – PhD
    Apr 27, 2023 at 17:08
  • $\begingroup$ The average second-order finite differences describe the typical direction that the curve is bending away from a straight line, which is a form of curvature. This is different than the curve either tending to increase or decrease, which is better described by the first-order finite difference. Consider generating some data for a line, exponential, and logarithm. You'll see that they have different signs for the average second difference, but the same sign of the average first differences. Those functions are all increasing. As an exercise, try out some more functions that are decreasing. $\endgroup$
    – Galen
    Apr 27, 2023 at 17:36
  • $\begingroup$ Now I understand. Rather well. Thank you. The first difference does capture the general "increment/decrement" and I have already tried those curves specifically. I'm trying to build an intuition for such differential averages TBH and I don't think I know enough to accurately articulate the interpretation. I'm more interested in "the rate of the rate of growth" (which implies the 2nd derivative i.e. acceleration from my physics days). If the computation can help me identify this, it matters more to me than just +ve trend per my example in the question. $\endgroup$
    – PhD
    Apr 27, 2023 at 21:17
  • 2
    $\begingroup$ The emphasis I have on curvature comes from its important roles in differential geometry. Even in discrete differential geometry it becomes important to compute analogs of curvature. $\endgroup$
    – Galen
    Apr 28, 2023 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.