Smart Statistics vs Unsupervised Learning for Anomaly Detection

Question

I'm a working on an Anomaly Detection project for college. Please do help me understand some concepts. Some of my biggest questions are:

1. How do we decide for a project on Anomaly Detection, should we just use sort of smart statistics or unsupervised learning? (I will be hoping to use the stats/model for newer data that I haven't seen before)
2. In a way, aren't smart statistics unsupervised learning itself? (You use the stat logic to build the model? )
3. How do you decide which variables are relevant for your model and classify it as univariate or multivariate?
4. I am also trying to understand the process of cross-validation. What I have so far understood is that you just try out different models and decide which one is the best. Isn't that just building different models? How does the process of cross-validation actually help me narrow down the subset of models that I am supposed to try and test?

Thanks!

Answer 1

1-3 : Many things are possible, from unsupervised machine learning to stats : you can make really simple things (put thresholds by your own, calculate point distance from neighbours once you normalized them, ...) or a bit more complicated (clustering and isolate less populated clusters for example, Isolation Forest algorithm). In unsupervised learning, chosing variables is way more subjective : you don't use metrics to show if your model is good / bad as you'd do in supervised (actually you can, but that's harder, I won't go into details). So that's a bit arbitrary : select variables you want to find the anomalies about, and try showing results and adapt since you like the result.

4 : Cross-Validation is a process (mainly used for Supervised) to evaluate your model. You have to understand one thing : when you create a model, you often separate 3 sets : Train, Test, Validation : Train is the set you use to train your model, and then you test on your Test. To make your model better, you adapt parameters and variables (which variables and how you present them) on train and test, to see how your metric (applied on how your model predicts your test) evolves. Once you have a correct model, you finally apply it on Validation. Validation is like a 2nd test set, that you didn't use to find the "best" model, so you're not biased. It gives you an overlook on how the model will perform on new, unknown data.

Cross validation is used when you don't want to split data : if you allow 60% for train, 20% for test and 20% for val, you actually are only training your model on 60%, which can be a problem (in case you don't have a lot of data for example). So you use Cross Validation. Let's say you use 10 folds : Split your data in 2 sets : Train (80%) and Validation (20%). Then, to tune your model and find the best for you, you'll only use your Train set like that : Cross Validation will cut it in 10, and use 9/10 as train and 1/10 as test. It then does that 10 times, changing each time the 1/10 used for test. After that, make the mean of your 10 results for your metric. Like that, you Trained and Tested your model on all your 80%, without splitting it again. Once you made the perfect model for you, test on your unused Validation to have an idea how it performs in "real world"