# Data mining algorithm suggestion

I would like to use data mining to try to find a good workout schemes. The input dataset will contain the parameters of a set of workouts with dates and different performance and medical measures. The problem is that the influence of each individual workout will be different depending on the time that has passed after this workout. For instance on the next day after the workout the performance will degrade, but eventually will improve to the higher level than before the workout. So each performance measure will be the cumulative result of different workouts. I know that I can run a series of workouts of the same type during a period of time and then use difference in the performance as a predictor. But I was wondering if there are any algorithm that allow to take into account the time component and run analysis that will take into account individual workouts.

• @Max It seems like a good question. I'm not sure about the "data-mining" tag, though. I'd suspect that there should be some kind of appropriate statistical (parametric) model for the data and question at hand. – chl Nov 1 '10 at 13:05
• @chl I tagged it data-mining because I was thinking about finding some rules and patterns in the data. Wouldn't that make it a data mining technique? The workout is just an example. In general I would like to know if there exist some algorithm that can deal with temporal cause-and-effect data. But I also would like to know if there's some specific statistical model for this particular example. – Max Nov 1 '10 at 14:10
• @Max A simple initial approach is to posit an exponential decay in effectiveness over time. The decay rate becomes a parameter in a model relating cumulative workout effect (as weighted by the decay) to outcome measures. If the decay is not huge and not zero and you have sufficient data you can move on to exploring other forms of decay. This is really in the spirit of exploratory data analysis. – whuber Nov 1 '10 at 14:45
• @Max How large is your dataset? If it's relatively small, what you're doing might be better characterized as "exploratory data analysis" rather than "data mining." – whuber Nov 1 '10 at 14:46
• @whuber Depending on how the data is presented, the dataset is from several tens to about a hundred records. – Max Nov 1 '10 at 15:59

Your first task is to find a reasonable model relating an outcome $Y$ to the sequence of workouts that preceded it. One might start by supposing that the outcome depends quite generally on a linear combination of time-weighted workout efforts $X$, but such a model would be unidentifiable (from having more parameters than data points). One popular simplification is to suppose that the "influence" of a workout at time $t$ on the outcome at time $s$ is

a. proportional to the intensity of the workout,

b. decays exponentially; that is, is reduced by a factor $\exp(\theta(t-s)))$ for some unknown decay rate $\theta$, and

c. independently adds to the influences of all other workouts preceding time $t$.

Of course we must be prepared to allow some deviation between the actual outcome and that predicted by the model; it is natural to model that deviation as a set of independent random variables of zero mean.

This leads to a formal model which can serve as a useful point of departure for EDA. To write it down, let the times be $t_1 \lt t_2 \lt \ldots \lt t_n$ with corresponding workout intensities $x_1, x_2, \ldots, x_n$ and let the outcomes be measured at times $s_1 \lt s_2 \lt \ldots \lt s_m$ with values $y_1, \ldots, y_m$, respectively. The model is

$$y_j =\alpha + \beta \exp(-\theta(s_j - t_{k})) \left( x_{k} + \exp(-\theta \Delta_{k,k-1}) x_{k-1} + \cdots + \exp(-\theta \Delta_{k, 1})x_1 \right) + \epsilon_j$$

where $\alpha$ and $\beta$ are coefficients in a linear relation, $k$ is the index of the most recent workout preceding time $s_j$, $\Delta_{i,j} = t_i - t_j$ is the time elapsed between the $i^\text{th}$ and $j^\text{th}$ workouts, and the $\epsilon_j$ are independent random variables with zero expectations.

This can get messy when workouts and endpoint measurements are unevenly spaced. If to a good approximation the spacing between a workout and the next measurement is constant (say, a time difference of $s$) and--as an expository simplification--if each workout is followed by a measurement (so that $m = n$), then this model suggests some useful EDA procedures. As an abbreviation, let's write (somewhat loosely)

$$f_k(x,t,\theta) = \left( x_{k} + \exp(-\theta \Delta_{k,k-1}) x_{k-1} + \cdots + \exp(-\theta \Delta_{k, 1})x_1 \right)$$

for the weighted sum of the workouts up to and including the $k^\text{th}$ one, whence

$$y_k = \alpha + \gamma f_k(x,t,\theta) + \epsilon_k$$

where $\gamma = \beta \exp(-\theta s)$. Note that this formulation accommodates any irregular time sequence of workouts, so it's not grossly oversimplified.

What you want to know is whether this makes sense: do the data at all behave like this? We're really asking about the possibility of a linear relationship between the $x$'s and the $y$'s. We need that to hold for at least one decay constant $\theta$ with a reasonable value.

One way to check is to note there is a relatively simple relationship between successive terms $f_{k+1}$ and $f_k$; you let $f_k$ decay for additional time $t_{k+1} - t_k$ and add $x_{k+1}$ to it:

$$f_{k+1}(x,t,\theta) = x_{k+1} + \exp(-\theta \Delta_{k+1,k}) f_k(x,t,\theta).$$

(This formula, by the way, provides an efficient way to compute all the $f_k$ by starting at $f_1 = x_1$ and continuing recursively for $k = 2, 3, \ldots, n$--a simple spreadsheet formula. It is a generalization of the weighted running averages used extensively in financial analysis.)

Equivalently, we can isolate $x_k$ by subtracting the right hand term. This suggests exploring the relationship between the adjusted values $z_k = y_{k+1} - \exp(-\theta \Delta_{k+1,k})y_k$ and the workouts $x_k$, because

$$z_k = (1 - \exp(-\theta \Delta_{k+1,k}))\alpha + \gamma x_k + (\epsilon_{k+1} - \exp(-\theta \Delta_{k+1,k}))\epsilon_k).$$

If the workouts are approximately regularly spaced, so that $\Delta_{k+1,k}$ is roughly constant, then for any fixed value of $\theta$ this expression is in the form

$$z = \text{ constant + constant *} x + \text{ error}.$$

The error terms will be positively correlated (in pairs) but still unbiased. It is now clear how to check linearity: Pick a trial value for $\theta$, compute the $z$'s (which depend on it), make a scatterplot of the $z$'s versus the $x$'s, and look for linearity. Vary $\theta$ interactively to search for linear-looking scatterplots. If you can produce one, you already have a reasonable estimate of $\theta$. You can then estimate the other parameters ($\alpha$ and $\beta$) if you like. If you cannot produce a linear scatterplot, use standard EDA techniques to re-express the variables (the $x$'s and $y$'s) until you can. Look for a value of $\theta$ that minimizes the typical sizes of the residuals: that is a rough estimate of the decay rate.

I don't expect this method to be highly stable: there is likely a wide range of values of $\theta$ that will induce linearity and relatively small residuals in the scatterplot. But that is something for you to find out. If you discover that only a narrow range of values accomplishes this, then you can have confidence that the decay effect is there and can be estimated. Use maximum likelihood; it will be convenient to suppose the $\epsilon$'s are normally distributed. (The profile likelihood, with $\theta$ fixed, is an ordinary least squares problem, so it will be easy to fit this model. Alternatively you could try fitting the relationship between $z$ and $x$ directly using generalized least squares, but I think that would be trickier to implement.)

This all might sound complicated but it's actually quite simple. You could set up a spreadsheet in which $\theta$ is the value in a cell, add a $\theta$-varying control to a scatterplot of $x$ and $z$ (computed in the spreadsheet from a column of $t$ values and a column of $y$ values), and simply adjust the control to straighten out the plot.

It will be harder to explore a dataset in which there are fewer (or more) measurements $y_j$ than there are workouts $x_k$ or where the temporal spacing between workouts and measurements varies a lot. You might have to settle for maximum likelihood solutions alone, without the benefit of supporting graphics to verify the reasonableness and the adequacy of this model a priori.

Even if my assumptions do not agree with your situation in all details, I hope that this discussion at least suggests effective approaches for furthering your investigation.

• I see one problem with this model. It is assumed that the effect of a workout is positive and descreases with the time. In fact at first the effect is negative, then it moves to a positive range and then decreases to zero. Other than that I believe that this procedure can generate some interesting results. Thanks. – Max Nov 1 '10 at 19:46
• @Max It is assumed only that the effect, whether positive or negative, decays at a constant rate over time. If you are sure that the change in effect over time is more complex than that, you can incorporate that assumption. (E.g., Y could be the sum of two terms [presumably of opposite signs] having two different decay rates.) This will introduce additional parameters that play the role of \theta and it could complicate the model sufficiently that you might have to rely on ML methods. I think you'll find you need lots of data! – whuber Nov 1 '10 at 19:56