# Time series analysis (removing effect of external factor)

I'm currently working on detecting the response from a sensor with the following profile:

The sensor responds to temperature fluctuations and I was wondering if y'all could suggest methods to "remove" the effect of temperature from the raw data so that we are only left the actual response from the the stimulus of interest. There is a about 1 minute delay between temperature change and it's effect on the sensor (inversely correlated too)

Raw data on: http://pastebin.com/hE30KgWg

• Can you post the data? – forecaster Jul 8 '15 at 17:50
• just added link to raw data. thx – ananio Jul 8 '15 at 19:25
• In looking at the data, I'm thinking of Transfer function modeling. What is the purpose of the whole analysis,is it prediction, identifying cause and effect ? – forecaster Jul 8 '15 at 19:40
• the goal is to detect the circled response in the most reliable way - in such a way that temperature change (which is out our control) doesn't cause any false positive. For now I'm using double exponential smoothing to predict the sensor values and compare it to the actual value and use that difference to decide whether the desired incident occurred or not – ananio Jul 8 '15 at 21:18

I used JMP.

I know the temp spike is an outlier.

Approach:

1. normalize the variables because the magnitudes make relative error less meaningful.
2. make 3 lagged variables (you have a very high sample rate). First is one-row lag of sensor, second is 10-row lag, third is 100-row lag.
3. exclude the known bad rows (~325 of 14857 rows)
4. use NN tool with 5 hidden nodes, 5-fold validation
5. range of error on known-good is [-53.6% to 56.6%] while range of error on whole domain is [-300% to 541%]. The known-bad values are way outside the NN.

How to use this:

• run NN to predict temperature values
• cull values where predicted value is worse than ~ +/- 50% of actual.

Best of luck

Here is the distribution of fit-error on known-good:

Here is the distribution of all errors (red is known-bad):

Here is JSL script for the NN:

Neural(
Y( :norm_temp ),
X( :norm_sensor, :nSens_lag1, :nSens_lag10, :nSens_lag100 ),
Missing Value Coding( 0 ),
Validation Method( KFold, 5 ),
Fit( NTanH( 5 ) )
);


Here is JSL for the lag-columns

Lag( :norm_sensor, 1 )
Lag( :norm_sensor, 10 )
Lag( :norm_sensor, 100 )


Here is the actual formula for the NN, inside the column. This isn't what generates it, it is the result of the generation process.

T#7 = :norm_sensor || :nSens_lag1 || :nSens_lag10 || :nSens_lag100 || [1];
T#8 = TanH(
0.5 * T#7 * [30.7473692662203, 29.0993763760375, -56.4559028410454, -
2.67448115098517, 1.30908183173594]
) || TanH(
0.5 * T#7 * [22.5759315516307, 8.89678971646283, -54.711022282694,
53.9236422700948, -16.6476167476592]
) || TanH(
0.5 * T#7 * [6.19157855914693, 18.6514641930598, -11.2582597431591, -
9.27557430692944, 4.19660893051184]
) || TanH(
0.5 * T#7 * [-41.2936201375412, -39.6395228784143, 78.3090529304216,
2.09734688960516, -1.49173365453039]
) || TanH(
0.5 * T#7 * [19.8007752288949, 6.88007350814404, -54.6160603867517,
59.0876374448871, -16.9587636515882]
) || 1;
T#8 * [-7.02732394844009, 4.36906102691856, 13.1114705112636, -4.99133519146221, -
4.17890676665167, -10.9695165376626];