This is a reasonable question but it might be conflating 2 things.
- Should $k$ be chosen to make [something] representative?
- Should the-way-we-assign-data-to-each-fold be chosen to make [something] representative?
1) Choice of $k$
In a comment the OP says "I read it here (the 'configuration of k' section)..."
And in that linked blog post, that blog's author writes:
Representative: The value for k is chosen such that each train/test group of data samples is large enough to be statistically representative of the broader dataset.
Rather than being representative of the broader dataset, it's often more useful to think about being representative of the population from which the dataset came.
That's because cross validation is less suited to answering "How well would this model perform on new data if I fit it to this dataset?" (conditional test error), and better suited to answering "How well does this model tend to perform on new data after being fit to datasets like this one?" (expected test error)
See for instance Section 7.12 of The Elements of Statistical Learning (Hastie et al., 2nd ed), or the preprint "Cross-validation: what does it estimate and how well does it do it?" (Bates et al., 2021).
Of course, often we still want to use the CV estimates to choose a model for our dataset. It's just helpful to realize that they are one step removed from directly answering that question.
For the moment let's just talk about the case when the original dataset was an iid sample from the population. No temporal correlation, group-hierarchical structure, etc.
In that spirit, if you choose $k=n$ and use LOOCV, each CV training set will be almost the same size as your full dataset. So CV will be "representative" in that it's trying to measure your model's expected performance when trained on datasets of (nearly) the same size as your full dataset. If that is indeed your goal, LOOCV is "representative" in that model performance estimates should be nearly unbiased.
By contrast, smaller $k$ such as $k=5$ might be "less representative" in that CV will try to measure your model's expected performance when trained on datasets smaller than the one you really have. Imagine you could get a plot of true expected model performance vs sample size. If this "learning curve" is still steep at sample size $n*4/5$, you'll get a pessimistic view of how well your model would tend to perform on samples of size $n$.
That's all in terms of repeated sampling: if you repeated the process of [collect an entirely new dataset of size $n$ and run CV] many times, LOOCV should have smaller bias than other values of $k$. But the standard story is that there's a bias-variance tradeoff, and LOOCV may have higher variance than others. So we often choose a moderate value like $k=10$ and hope that the learning curve is flat enough for samples of size $n*9/10$ to be "representative" but not too variable across repeated sampling.
In any case, in a complete run of $k$-fold CV (using all the folds), every observation will be in a test fold exactly once. Different values of $k$ will make those test cases more or less correlated with other observations (affecting the variance over repeated sampling)---but won't change the total number of test cases. However, if you're only doing a single train/test split, then yes, you should try to get enough data in both the training set (so you can afford to fit realistically/usefully complex models) and in the test set (so you can evaluate them with enough precision to distinguish better models from worse ones).
2) Choice of how to assign data to each fold
The OP also writes:
...should be picked in such a manner as to have representative validation sets (folds). This seems to me to be contradictory since the leave-one-out CV has only one sample as a fold, which clearly can't be representative of the dataset...
That's okay. If the dataset is iid from the population, any one test case is a "representative" sample ($n=1$) of that population. And the set of all LOOCV test cases across folds is representative of the population too.
This relates to a common misconception:
Shouldn't we try to make each fold look as similar to the full dataset as possible?
and I would answer:
No. In the spirit of conditional vs expected test error, the way each fold is sampled should be representative of how the full dataset was sampled.
Reference: "K-fold cross-validation for complex sample surveys" (Wieczorek et al., 2022).
This will help CV to estimate "How well is this model expected to perform on datasets collected the same way as this one?" Otherwise, if CV ignores something important about your sampling design, it will be a much less useful approximation to "How well might this model work on my full dataset?"
So for instance, CV for time-series data generally keeps blocks of temporally-consecutive observations together, and also ensures that we train on earlier data and test on later data. If we formed folds randomly instead, the training data might look "more representative" of the full dataset---but it wouldn't be useful for assessing our ability to make forecasts.
Reference: "Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure" (Roberts et al., 2016).
Likewise, in CV for grouped data (like in @Henry's comment), generally we want to keep entire groups together when making CV folds. If we have several observations for each subject in a study, folds should be formed subject-by-subject since this is how we sampled the data. (So $k$-fold would put $n/k$ subjects in each fold, and LOOCV would put one subject in each fold.) If the training and test sets have different subjects, this lets us honestly evaluate how well we'd make predictions for future subjects not yet in the dataset.
Reference: "The need to approximate the use-case in clinical machine learning" (Saeb et al., 2017).
Finally, since stratified CV came up in the comments: I would not usually do stratified CV unless the original data collection really was stratified. Otherwise CV may be over-optimistic. It will be estimating "this model's performance when trained on stratified data," whereas you actually want an estimate "...when trained on iid data" or whatever.
(The exception of course is a classification problem with a rare outcome category: then you might be forced to stratify on the response, because otherwise you might never see that outcome in some training sets.)
