Find Best Model Parameters With K-Fold Cross Validation

Question

I have a word prediction model where you enter 2 or more words and it predicts the next word. This model (Katz Backoff Trigram) consists of 3 tables: t1, t2, and t3 (unigram, bigram, and trigram frequency tables), and 2 parameters $\gamma_2$ and $\gamma_3$ (bigram and trigram discount rates). The tables are built from a corpus of roughly 30M words of which roughly 1M are unique. The goal is to determine the best values for the 2 parameters $\gamma_2$ and $\gamma_3$ that optimize the accuracy of prediction.

I'm doing a 5-fold cross-validation to determine the best values for these parameters as well as estimate the test error. I start by creating 5 folds of roughly equal size depicted as the rows in the figure below using random sampling without replacement:

20% of the data within each fold is set aside as a validation set and is depicted in white. Because my model is a classifier, I need to further split the non-validation data into two additional partitions depicted in blue (80%) and gold (20%).

The parameters $\gamma_2$ and $\gamma_3$ range from 0.1 to 1.9 inclusive incremented by 0.1, so I create a 19 x 19 grid of each combination ($\gamma_2$, $\gamma_3$). Then, for each fold, I create t1, t2, and t3 from the blue data and pull 500 random trigram samples from the gold data to make predictions on. For each ($\gamma_2$, $\gamma_3$) pair, I extract the first 2 words of the random trigrams and have the model predict the last word. The successes are tallied and used to calculate the accuracy. The output csv file looks something like this:

"gamma2","gamma3","predict","success","acc"
0.1,0.1,500,91,0.182
0.1,0.2,500,92,0.184
...
1.9,1.8,500,102,0.204
1.9,1.9,500,101,0.202

My question is procedural/process in nature. I have 5 sets of output csv files like the one shown above. How do I select the best ($\gamma_2$, $\gamma_3$) pair?

My understanding is that I should take the ($\gamma_2$, $\gamma_3$) pair that gives the highest accuracy in each fold and run it on the validation set, then take the pair that give the highest accuracy from each of the 5 validation sets.

I could then get an estimate of the test accuracy by averaging the accuracy of the best pairs ran on each of the validation sets.

Does this make sense? If so, my last question has to do with the fact that I get several ($\gamma_2$, $\gamma_3$) pairs that give the highest accuracy in each fold. My intuition is to look for overlapping pairs across the folds (using something like an aggregate heat map) and select the parameters this way. Is this a valid approach?

Answer 1

Usually language models are not evaluated using an accuracy metric. That is for two reasons. One is that the accuracy will often be quite low due to the large variety of possible next words to choose from. Second, most language model applications don't require you to make a hard prediction about which word will come next. Instead they expect a probability distribution over the possible next words. A better way of picking the best parameter setting is to choose the one that gives the best average log probability for the next word.