Can a 2/3 training 1/3 test split be unrepresentative? We often fit a predictive model using 2/3 of the data and assess fit on the remaining 1/3. Can such a split be unusual? Is it best to average over many such splits? I realize that this would produce potentially many models. But, in the related approach of cross validation, such repeated splits are advised. In that case, we fit a final model and use the results of CV as a measure of predictive ability.
 A: Test set performance is an estimator for the learning algorithm's performance on future data. We'd like the estimate to be as close as possible to the 'true' value (i.e. what we'd see if we had access to the underlying distribution). The error of an estimator can be expressed in terms of its bias and variance. Bias measures the systematic difference between the estimate and the true value, and variance measures the extent to which the estimate fluctuates (across datasets drawn from the underlying distribution). One seeks to reduce the error by reducing the bias and variance, but there's typically a tradeoff between them.
Say we're using simple holdout (e.g. a single training/test set split). As the number of points in the test set shrinks, the variance of the performance estimate increases because it becomes more susceptible to the points that happened to fall in the test set by chance. One could try to compensate for this by increasing the fraction of points allocated to the test set. But, this would shrink the size of the training set, giving a poorer model fit and thus increasing the bias.
Instead, one could perform multiple training/test splits and average the test performance. Different variants of this procedure exist (e.g. k-fold cross validation, monte carlo cross validation, repeated k-fold cross validation, plus related techniques like bootstrap). Averaging over multiple test sets reduces the variance. This gives some wiggle room for allocating a larger fraction of points to the training set on each iteration, reducing the bias. The downside of this approach is that the model must be fit multiple times, increasing the computational load.
Typically, simple holdout is used when the dataset is large--there are enough points that a model can reasonably be fit on the training set and evaluated on the test set, and perhaps too many points for cross validation to be computationally feasible. Otherwise, a technique like cross validation is a better approach.
K-fold cross validation (e.g. 10 fold) or (repeated k-fold cross validation if you have the computational resources) is probably better than repeating a simple 2/3 vs. 1/3 split. You can find articles comparing different variants of cross validation.
