We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can span hundreds of tokens. We've defined a cutoff at 256, but creating batches randomly and then padding is quite costly in terms of training time. We are now sorting the posts by length and then sampling randomly in consecutive blocks of n posts, where n is the maximum number of posts of 256 tokens that we can fit in a batch without running out of memory. But for batches of smaller posts (like n posts of length 2), this is not utilizing the resources we have and our compute time is quite limited.
So now we're thinking we could pack sequences together in batches such that n*length(post) remains roughly constant across batches. So one batch would be 10 sequences of length 256, while another would be 1280 sequences of length 2. But we're not sure what impact this will have on the training. It seems the learning rate is now scaled to tokens rather than posts, which actually sounds desirable with this kind of variance in number of tokens, but maybe not? Does the learning rate need to be adjusted somehow?
I've seen others do something similar called "tensor packing", where you would concatenate all the posts and then chop them into chunks of the same size. I don't like the idea of combining posts that have nothing to do with each other like that (so the model could learn predict one from the other), but other than that this seems to do roughly the same thing regarding the learning process and the learning rate, right?
