Time series analysis (removing effect of external factor)

Question

I'm currently working on detecting the response from a sensor with the following profile:

The sensor responds to temperature fluctuations and I was wondering if y'all could suggest methods to "remove" the effect of temperature from the raw data so that we are only left the actual response from the the stimulus of interest. There is a about 1 minute delay between temperature change and it's effect on the sensor (inversely correlated too)

Raw data on: http://pastebin.com/hE30KgWg

Answer 1

I used JMP.

I know the temp spike is an outlier.

Approach:

1. normalize the variables because the magnitudes make relative error less meaningful.
2. make 3 lagged variables (you have a very high sample rate). First is one-row lag of sensor, second is 10-row lag, third is 100-row lag.
3. exclude the known bad rows (~325 of 14857 rows)
4. use NN tool with 5 hidden nodes, 5-fold validation
5. range of error on known-good is [-53.6% to 56.6%] while range of error on whole domain is [-300% to 541%]. The known-bad values are way outside the NN.

How to use this:

• run NN to predict temperature values
• cull values where predicted value is worse than ~ +/- 50% of actual.

Best of luck

Here is the distribution of fit-error on known-good:

Here is the distribution of all errors (red is known-bad):

Here is JSL script for the NN:

Neural(
    Y( :norm_temp ),
    X( :norm_sensor, :nSens_lag1, :nSens_lag10, :nSens_lag100 ),
    Missing Value Coding( 0 ),
    Validation Method( KFold, 5 ),
    Fit( NTanH( 5 ) )
);

Here is JSL for the lag-columns

Lag( :norm_sensor, 1 )
Lag( :norm_sensor, 10 )
Lag( :norm_sensor, 100 )

Here is the actual formula for the NN, inside the column. This isn't what generates it, it is the result of the generation process.

T#7 = :norm_sensor || :nSens_lag1 || :nSens_lag10 || :nSens_lag100 || [1];
T#8 = TanH(
    0.5 * T#7 * [30.7473692662203, 29.0993763760375, -56.4559028410454, -
    2.67448115098517, 1.30908183173594]
) || TanH(
    0.5 * T#7 * [22.5759315516307, 8.89678971646283, -54.711022282694,
    53.9236422700948, -16.6476167476592]
) || TanH(
    0.5 * T#7 * [6.19157855914693, 18.6514641930598, -11.2582597431591, -
    9.27557430692944, 4.19660893051184]
) || TanH(
    0.5 * T#7 * [-41.2936201375412, -39.6395228784143, 78.3090529304216,
    2.09734688960516, -1.49173365453039]
) || TanH(
    0.5 * T#7 * [19.8007752288949, 6.88007350814404, -54.6160603867517,
    59.0876374448871, -16.9587636515882]
) || 1;
T#8 * [-7.02732394844009, 4.36906102691856, 13.1114705112636, -4.99133519146221, -
4.17890676665167, -10.9695165376626];