I have watched Andrew Ng's lecture Error Analysis and the first slide of the lecture says:

Error analysis: Manually examine the examples (in cross validation set) that your algorithm made errors on. See if you spot any systematic trend in what type of examples it is making errors on.

I believe that the term "cross validation set" here has nothing to do with cross validation technique as he has never mentioned it before this lecture, so the "cross validation set" here is just a normal validation set if I'm not misunderstanding (or maybe "cross validation set" = "validation set"? I'm a newbie to Machine Learning by the way...).

My main question is: Doesn't that make me do data snooping on the validation set, which will make the validation error of the hypothesis with least validation error much less accurate than it should be?


I was watching the videos just now and I had the same feeling as you did.

Coming back to the question, yes, you are right that the 'cross validation' dataset Andrew refers to is in fact the normal validation set, without any cross validation techniques such as K-Fold involved.

In practice, when conducting error analysis or plotting the learning curve for certain algorithms, it is always recommended to analyze the results obtained from the CV techniques, since it 1) spare the dataset from validation set thus we have more training data 2) lower the variance (less prone to overfitting) since we average out the cross validation errors.

Python user: Here are some recommended sklearn functions and packages that I found useful to me, have a look if they interest you:

Learning curve: http://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html Tips: You can modify the function a little bit to use the scorer that you prefer. The sklearn make_scorer() package is also useful to make metrics into scorer if the corresponding scoring does not exist.

SelectFromModel: http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html Tips: I found it an out-of-box feature selection solution which works fairly on any model, since the method selects features based on the metadata of the model. Particularly for lowering variance.

Polynomial features:

Out-of-box method for generating higher degree polynomial features, recommended to generate a huge set of features with this package first and then use the SelectFromModel.

  • $\begingroup$ I'm sorry for the late upvote and the late reply as I have had to re-read your answer so many times to understand what you mean (it is my brain's fault, not yours). Thank you for your suggestions. I did not make yours as an answer as I am looking forward to a reasonable couterargument or an answer that truly opens my mind about Andrew Ng's reason, I am very sorry about this. $\endgroup$ – ntvy95 Apr 7 '17 at 9:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.