Checking cross-validation accuracy using regression I want to empirically check cross-validation accuracy. That is, I am trying to check and better understand bias and variance in the estimation of error by varying the $K$s in K-Fold. For this purpose I need the real error of the model.
My idea is to use this function:
$$y(x) = \log{x}\sin{x} + 1/10N(0, 2\log(1+x)) + 2\cos{x}$$
to generate a random sample $D$ and then use cross-validation techniques for this sample $D$ and a regression model (simply a polynomial of degree $10$) to estimate the error. Then I could compare this to the error obtained by using the model to a very large sample generated from the function above.
I would do this because for example neural networks are computationally expensive. Is this approach reasonable?
 A: I think this is reasonable for assessing CV error in this particular case, without generalization to other models and datasets.  This method is applied in ISLR and shown in Fig 5.6. 
 
Here the authors simulate data and therefore can compute the true MSE.  One thing they show is that even though CV might not be equal to the true error (it's not unbiased---in this case, it underestimates the error), the CV estimate tends to be around a minimum when the true error is also around a minimum, which is why CV is very good for model selection. 
Keep in mind however that your "check" of cross validation would only be valid for this particular dataset/model pair.  I guess I am still not sure what your goal is with respect to the neural network---perhaps to see which K is best using this synthetic data so then you know which one to use for your neural network, without having to essentially do a search over K using the neural network.  I don't think this procedure will give you this information.
In practice, K in the range 5 to 10 is commonly used. Consider the extremes: LOOCV leads to high variance and low bias of the estimate; very large K leads to high variance of the estimator (Imagine with an N observation dataset choosing K=N-1. For each fold, you then train on only one case and, unless your data is very bland, get a completely different estimator for each one) along with lower variance and higher bias of the estimate. K equal to 5 to 10 seems to be a good settling point.  To quote ISLR:

as these values have been shown empirically to yield test error rate estimates that suffer neither from excessively high bias nor from very high variance

