0
$\begingroup$

i'm working on dataset contain machinery sensor data. each column(feature) represent different sensor data(pressure, temperature, speed, etc) of the machine part. here task is to predict normal behaviour of machine when input(actual sensor data) contain noise.

i'm using auto-encoder here. my dataset contain features with different scale so feature scaling is necessary for this task. I tried MinMax and Standard scaler. Standard scaler works best with tanh activation function.

but there is problem with scaling. when noise value is very high or most of data points are anomalous(70%-90%),then prediction follow noise. for example when I use standard scaler for feature scaling and I added some noise value(delta=100) to 80% of the data. it changes mean and standard deviation of the data. these two variables are use to scale input. Also use to inverse scaling for prediction. so final result is affected by this mean and it follows noise instead of giving normal output. so what is proper way to do feature scaling for this kind of task ?

case 1: following picture show when I add delta=300 to 90% of actual data and perform standard scaling on it. picture shows that model consider noise(0-1000) as normal data and normal(1000-1200) as noisy data

**case 1:**following picture show when I add delta=300 to 90% of actual data and perform standard scaling on it. picture shows that model consider noise(0-1000) as normal data and normal(1000-1200) as noisy data

case 2: following picture show scaling result of actual data when I add very large delta=3000 to data-points(0-200). this affect mean of the data so scaling follow noise in this case also.

**case 2:** following picture show scaling result of actual data when I add very large delta=3000 to data-points(0-200). this affect mean of the data so scaling follow noise in this case also.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.