6
$\begingroup$

I work on a method that gives a (noisy) estimation of brain volume over time in Alzheimer's patients.

As we know that the evolution is smooth and even mostly linear if looked at over a time frame of a few years, one way of evaluating the algorithm is to look at the smoothness or linearity of the estimated time series (brain volume over time).

This is why I'm looking for a good metric of smoothness or linearity of time series. What I've been using for now is the average R² from patient-wise linear regressions. This does reflect linearity, but it depends heavily on the sampling (it will give better results with fewer data points). Could anyone suggest a better metric?

Some details since apparently my question wasn't clear. I have an algorithm that gives a few (from one to five or six) discrete $V(t_n)$ brain volume measurements for many patients. I want to measure to which degree $V$ is close to something linear, eg. $V(t_n) = a t_n + b$ accross all patients (all patients will have different coefficients, but I want an overall measure of linearity).

I should also mention that the $t_n$ values are not regularly spaced.

One more comment Obviously patients with one or two data points are not informative. However I want to take advantage of all the other patients. Even a patient with three data points is informative: three aligned points are much better than three completely jagged points.

$\endgroup$
  • 2
    $\begingroup$ One possible measure of smoothness is mentioned here $\endgroup$ – Glen_b Sep 30 '13 at 10:48
  • $\begingroup$ @Glen_b : thanks for the link! I understand the regularly sampled solution, but I don't understand what these "divided differences" are for. Could you explain a bit more? $\endgroup$ – static_rtti Sep 30 '13 at 12:14
  • $\begingroup$ So you're taking other measurements and your algorithm calculates an estimated brain volume measurement based on those, and you want to check the realism of your algorithm by seeing if it is smooth and basically linear? I'm not sure that smoothness and linearity would actually indicate that your algorithm is reasonable, but you'd have to throw away any results with only one or two measurement points and only consider those with 3-6 points and talking about the smoothness of 3 points, for example, seems problematic. $\endgroup$ – Wayne Sep 30 '13 at 13:20
  • $\begingroup$ @Wayne: yes, that's basically it. Of course smoothness/linearity is only a tiny part of the picture, but it is a very good indicator of noise. I agree with throwing away patients with only one or two data points. My concern is that those with 3 and those with 6 should be treated in a way that does not favor one way or another. $\endgroup$ – static_rtti Sep 30 '13 at 13:27
3
+50
$\begingroup$

I think that your problem is simply overfitting - you want to have a linearity measure that isn't inflated at very small N.

Here are two candidates:

  1. Adjusted r-square. In your case (a single regressor), that would be $R^2-\frac{1-R^2}{N-2}$

  2. Cross-validated $R^2$: leave out one measurement at a time. Estimate the linear model without it. Then, calculate the squared difference between the actual held out measurement and its predicted value. Repeat for all time points, add up and you'll get predicted residual sums of squares (PRESS). Next calculate SST (the variance of the measurements * N). From here you can get a predicted $R^2$: $predicted \ R^2=1-\frac{PRESS}{SST}$

I think that the former tends to be optimistic whether the latter tends to be pessimistic. Both naturally require at least 3 measurements.

$\endgroup$
2
$\begingroup$

Although it would be better if you had described your problem mathematically, from what I read I understand that

a) You have an estimator function that estimates the evolution of brain volume over time, say, $\hat V_t= f($data$)$ and
b) You want to measure the distance of this estimator function from the simple linear time trend $\bar V_t = a + bt$.

So it appears that you should search the various distance measures available, like for example the Hellinger distance (which although it has been devised to measure the distance between two distributions, it can be used more generally to measure the distance between any two sets of points that can be linked pairwise).

But what I don't understand is this: why are you not satisfied with the, supported by the data as you write, simple linear estimator? Are you essentially trying to achieve a better short-term prediction performance, while maintaining as much "long-term linearity" as possible? Is this the case?

$\endgroup$
  • $\begingroup$ I'll try to make myself clearer. I can't give a mathematical formulation of the problem, because it's not a mathematical problem. The question is more what mathematical tool to use to handle the problem. $\endgroup$ – static_rtti Sep 30 '13 at 12:01
  • $\begingroup$ I think what wasn't clear is that my data is discrete. Basically my algorithm (let's forget about the word estimator which has a specific meaning in statistics) spits out a number of $V(t_n)$ values which should be approximately aligned. I'm looking for a measure of this linearity. $\endgroup$ – static_rtti Sep 30 '13 at 12:03
1
$\begingroup$

Since you are computing $r^2$ for each patient $i$ and you want the following properties:

So patients with more data points will likely have a worse metric, if nothing is done.

  1. Series with more measurements should be higher weighted

My concern is that those with 3 and those with 6 should be treated in a way that does not favor one way or another

  1. Series with lesser points should also be included judiciously.

A simple heurisic comes to mind to measure linearity across the whole dataset: Let the number of data points for each patient be $n_i$ and the total number of patients be $K$. Since you are already computing $r_i^2$ for each patient $i$, how about getting a weighted average: $$\frac{\sum_{i=1}^{K}n_i r_i^2}{\sum_{i=1}^{K}n_i}$$ This has the property that it will give more importance to series with higher number of observations. Instead of $n_i$ as the coefficients in the numerator and denominator above, you can use any other function of $n_i$ to address your concerns I quoted above.


Another thought: To counter the issue of ireegularly sampled observations,do the following. Fit $n_i-1$ dimensional polynomials (link1,link2) to each of the time series observations you have (as an extension of the divided differences idea by @Glen_b). Given $n_i$ algorithm outputs $\{V(t_k)\}$ at $n_i$ times $\{t_k\}$, this is just $$ V^{\textrm{poly. approx}}(t)=\sum_{k=1}^{n_i}V(t_k)\cdot\prod_{1\leq j\leq n,j\neq k}\frac{t-t_k}{t_k-t_j}. $$ Then, evaluate each of these polynomials at regular time locations for each patient. And then fit the lines and proceed as above.

$\endgroup$
0
$\begingroup$

I suggest two different approaches:

  • First of all, I have read that you suffered from sampling effects. One recommendation is using Wavelets/Fourier coefficients to obtain linearity. In many cases, there are solutions to avoid sampling effect with this tool.

  • Secondly, you should try kurtosis and skewness to see the variability of the signal and smoothness.

But, if you want to see trends on your signal, the best option is using Wavelet.

$\endgroup$
  • $\begingroup$ I have very few data points. See my edits, the question should be clearer. $\endgroup$ – static_rtti Sep 30 '13 at 12:07
0
$\begingroup$

How about doing your linear regression, then take the residual by subtracting your regression line from the data, and look at its standard deviation (SD) or median absolute deviation (MAD)? (The SD part may be basically equivalent to the R^2, I'm not sure of the math.)

The problem is the "one to five or six" part. That's not much data there, but of course the data you have is what you have. (If you had a lot more data, you could consider decomposition methods on the residual to look at the power in higher and lower frequencies.)

I thought of things like the KL-divergence or something like Alecos mentioned, but they're for differences in distributions not differences in time series, so I'm not sure they're applicable.

$\endgroup$
  • $\begingroup$ I think that's basically the same idea. What's for sure is that it doesn't adress the varying number of data points: two points will always be aligned, no matter what. Adding points makes the problem gradually more difficult. $\endgroup$ – static_rtti Sep 30 '13 at 12:55
  • $\begingroup$ @static_rtti: Your comment is a little puzzling. With one or two points you can't say anything about smoothness or linearity, by definition. Starting at three points, you can talk about it, but your answer won't have much meaning. The more points you have in your time series, the more meaningful "linear" or "smooth" will be. $\endgroup$ – Wayne Sep 30 '13 at 13:11
  • $\begingroup$ I mean that the more points you have, the less likely it is that there are aligned. So patients with more data points will likely have a worse metric, if nothing is done. That's what I'm worried about. $\endgroup$ – static_rtti Sep 30 '13 at 13:30
0
$\begingroup$

If you assume that the relationship is linear, you can use this trend estimator: $T_{n-1}=\frac{V(t_n)-V(t_{n-1})}{t_{n}-t_{n-1}}$ ,take the mean of these for each patient (which is the measure of linearity), calculate the sum of the residuals^2 (which gives an idea about how much you can trust this linearity. You can use Student t-test to measure the strength of the linearity in the same way it is used to test the strength of a trend in a linear regression in case you have a big enough sample size.

It is easier if you have a data sample we can use .

HTH

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.