validating model in machine learning - what does it mean in reality (intuition)

Question

Could someone explain (in simple way) what does mean of validating model ?
I tried to understand it, but I didn't managed to.

1. I can do cross-validation, but I am not sure about if it is validation.
2. What is the aim of validation ?
3. What are ways of validation ?

As I mentioned above - I don't expect very long expactation (that I can find in book), I have a problem with intuition here.

Answer 1

Usually, when you train your model, you split your dataset into two:

Train set

Train set is there for your model, so it can learn from it.

Test set

After your model is trained, you will show it data it has not seen before and see how well it will behave.

And now comes the validation part.

You are interested in how your model behaves in general. How well it can predict your target values. Is it poor estimation? Is it good one? And what if your model just learnt the pattern in training set (=overfitting).

So you must do validation on training set:

• What is the learning error?
• You do cross validation to see how it really behaves and that it does not overfit (check this question)

And you must do validation on test set:

• To be sure that your model just did not learn the train set because you want to catch the big picture, not copy your train set

When validating, you can use various measures depending on your situation:

Accuracy, Precision, Recall, ROC curve, AUC, F1, etc.