What is the benefit of k-fold cross validation? I understand that using 100% of the dataset and doing k-fold cross validation instead of train_test_split would eliminate that randomness the latter method have in splitting and thus potentially avoid overfitting.
But I have seen that this is not the best practice when it comes to k-fold CV. What I have seen is that we should split the dataset (say 80% training, 20% testing), and then perform k-fold CV on the 80% training (Image below).
What I am confused about is that with this method, we are again potenially falling into a random split by using train_test_split before CV. Why is this method then considered best practice generally ? What am I missing ?

 A: You can mend this by doing nested CV.
However... since the test data is only used to assess final model performance, not for deciding anything, one usually goes for the simple split.
A: K-fold cross validation is used to determine the general fit of a model for a modelling task. It is especially useful when the amount of data is limited.
Once you determine which model is best for your problem via k-fold CV, you will train the chosen model on the entire training set, and then test the model on a dataset that was never used during the modelling process. The test set should only be used once to prevent bias. This is why you need to split the data into a training and test set before k-fold CV. Otherwise you run into the problem of having a biased model evaluation.
A: Generally we would prefer to have more data, not less. So we need to make efficient use of our data. That can leave you with smallish test datasets. If you have one 20% test dataset you do not know, how precisely that describes the predictive value. If you have 5 different 20% test data Sets you can get a rough idea of whether these lead to 5 similar or 5 diverging result.
There is always leave-one-out on the other extreme but that gets computationally challenging very fast.
