4
$\begingroup$

I'm building a convolutional autoencoder as a means of Anomaly Detection for semiconductor machine sensor data - so every wafer processed is treated like an image (rows are time series values, columns are sensors) then I convolve in 1 dimension down thru time to extract features. I'm confused about the best way to normalise the data for this deep learning ie. if I normalise within each wafer I remove potential trends in the data - but from my trials I see that this results in trending Recon Errors. This ultimately makes it hard to create a detection threshold..? If I normalise across the entire training data set then the trends are present and recon errors can look different over time, but the loss during epochs are huge (like 1e13 instead of 0.xxxx when I normalise within a wafer run).

I'm wondering if someone can offer advice on the best way to go with this..?

P.

Improve this question
$\endgroup$
1
  • $\begingroup$ I haven't thought through the complete implications of the following suggestion, but what happens if you feed the autoencoder both types of normalized scores? My thought is that you would be providing both between-sensor and within-sensor measures of variation. $\endgroup$
    – Matt Barstead
    Commented Oct 27, 2019 at 13:22

1 Answer 1

Reset to default
1
$\begingroup$

I cannot underpin this mathematically but from my feeling you should standardscale each wafer individually because basically you are interested in them individually. When you scale by pool you take into account too much information. When you have to take into account trends I would advice to consider (also/in addition) a LSTM. LSTM-AEs are quite popular.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.