Comparing sensor data across multiple experiments

Question

I have some questions about doing statistics on a series of sensor measurements.

I have 12 series of data of sensor measurements (a force sensor, measuring at 10 Hz for 30 minutes). Each series consists of, say, 18000 measurements taken at fixed time points. Each series comes from an experiment in which a force was applied for the first 10 or so minutes, after which it remained constant.

6 experiments were performed using method "A" and 6 experiments were performed using method "B" (all with the same sensor), so a total of 12 experiments was performed.

So, for every time point t[0 ... 18000], there are 6 measurements for method "A" and 6 measurements for method "B". Do keep in mind that these 6+6=12 measurements were not acquired simultaneously, but in 12 separate experiments.

How would I go about finding out whether there is a significant difference in the measurements between method "A" and method "B"?

What I have thought of so far: I need to use a non-parametric test, because my measurements don't follow a normal distribution (a histogram confirms this). Also, these are all independent, non-paired samples. For every time point, I need to use the Wilcoxon rank-sum test (not signed), with 6 method "A" measurements and 6 method "B" measurements.

For example:

t = 60
A = [0.4, 0.5, 0.6, 0.5, 0.7, 0.8]
B = [1.0, 0.8, 0.7, 0.9, 1.0, 0.6]

-> Wilcoxon rank-sum test on (A, B) (= scipy.stats.ranksum)

Am I thinking in the right direction? FWIW I'm using Python/SciPy.

Answer 1

If each of the 12 experiments was done with the same application of force over time to the same type of object, then a linear regression that models both the change in time and the differences between methods A and B provides a useful approach.*

A linear regression can directly model the overall shape of the change in sensor output over time as it estimates the differences between sensors A and B. You perform a linear regression of sensor output against a cubic-spline-based representation of the change over time and a categorical factor representing which sensor was used. This makes no assumptions about the parametric form of the change of output over time, rather finding a representation of the change over time (coefficients for the spline terms) from the data.

If you specify the number of knots needed to capture the expected complexity of the change over time, some standard statistical software packages (I suspect including SciPy, but I don't use that myself) can fit the coefficients representing the splines as part of the linear regression. As you are imposing changes over the first 10 minutes and maintaining things constant thereafter, you could specify the locations of the knots in time rather than accepting the default values (typically evenly spaced) to capture the more complex behavior you might expect at early times.

The coefficients for the spline terms will provide a representation of the mean change over time. The statistical significance of the coefficient for the factor representing the sensors will test the mean A-B difference, with that mean change over time taken into account. This way, instead of doing 18000 tests you are doing a single test for the A-B difference. The spline approach only estimates a handful of coefficients from the data (one less than the number of knots), so you have a good deal of power to detect a difference. If you instead, say, treated each of the 18000 time points individually then the regression would be estimating about 18000 coefficients from the data and lower your ability to detect a true A-B difference.

This approach also provides a way to examine differences in the variance between A and B. For each, examine the distribution of differences between the observed values and those predicted from the regression model. Examine whether these differences depend systematically on the predicted values. If so, then you might want to adjust your model. For example, log-transformed sensor values might be better behaved than raw values if sensor errors are proportional to signal magnitude instead of independent of it. If observed-predicted differences don't depend on predicted values, then the variance of those differences provides an estimate of sensor variance (plus any among-experiment systematic variance and variance around the spline fit).

This approach doesn't, however, get to the issue of differences in time-course of responses between the 2 sensors. Sampling at 10 Hz suggests that you might be interested in changes as rapid as 5 Hz. The approach I suggest smooths out the changes over time and won't directly distinguish response time courses between A and B. The time course of your experiment, however, suggests that this isn't a primary issue for you.

---

*If there were systematic differences in the level of force applied or other aspects of the experiments, then you should include predictors in you linear regression model that account for those differences. A simple categorical factor representing the experiment would account for baseline shifts in sensor values among experiments. A categorical factor would work for fractional differences among sensor outputs if you work with log-transformed sensor data. More complicated situations might require including coefficients for interactions between experiments and the spline-based representations of time.