Manual error analysis on validation set? I have watched Andrew Ng's lecture Error Analysis and the first slide of the lecture says:

Error analysis: Manually examine the examples (in cross validation
  set) that your algorithm made errors on. See if you spot any
  systematic trend in what type of examples it is making errors on.

I believe that the term "cross validation set" here has nothing to do with cross validation technique as he has never mentioned it before this lecture, so the "cross validation set" here is just a normal validation set if I'm not misunderstanding (or maybe "cross validation set" = "validation set"? I'm a newbie to Machine Learning by the way...).
My main question is: Doesn't that make me do data snooping on the validation set, which will make the validation error of the hypothesis with least validation error much less accurate than it should be?
 A: I was watching the videos just now and I had the same feeling as you did. 
Coming back to the question, yes, you are right that the 'cross validation' dataset Andrew refers to is in fact the normal validation set, without any cross validation techniques such as K-Fold involved. 
In practice, when conducting error analysis or plotting the learning curve for certain algorithms, it is always recommended to analyze the results obtained from the CV techniques, since it 1) spare the dataset from validation set thus we have more training data 2) lower the variance (less prone to overfitting) since we average out the cross validation errors.
Python user:
Here are some recommended sklearn functions and packages that I found useful to me, have a look if they interest you:
Learning curve:
http://scikit-learn.org/stable/auto_examples/model_selection/plot_learning_curve.html
Tips: You can modify the function a little bit to use the scorer that you prefer. The sklearn make_scorer() package is also useful to make metrics into scorer if the corresponding scoring does not exist.
SelectFromModel:
http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html
Tips:
I found it an out-of-box feature selection solution which works fairly on any model, since the method selects features based on the metadata of the model. Particularly for lowering variance.
Polynomial features:
Out-of-box method for generating higher degree polynomial features, recommended to generate a huge set of features with this package first and then use the SelectFromModel.
