I have re-trained GPT2 model using over 10 million sentences for QA. And while testing also I am getting very good results. The only problem now I am facing is that I have millions of test data that I need to process at once. Referring to open-ai github, I have made few changes by looping over test sentences to get the prediction rather that getting output for one sentence at one time. Using CPU I am getting ~30sec for predicting output of each sentence. But we all know from a product perspective if we have millions of data, then this speed is not feasible. So how can I reduce inference timing of my model?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ GPT-2 is an absolutely massive model, and you're using a CPU. In fact, even using a Tesla T4 there are reports on Github that this is taking ms-scale time on batches of 10-100 docs (~60 tokens), which is well beneath your use case. There are, of course, a bunch of hardware considerations that could affect inference time -- but GPT-2 is massive, and I think this is a limitation of this model. You could try looking at a smaller language model (or some other architecture depending on your task) and trying to use that. $\endgroup$– chang_trentonCommented Jun 14, 2020 at 7:25
-
$\begingroup$ You can use Nvidia TensorRT library for faster inference: developer.nvidia.com/blog/… $\endgroup$– RaufCommented Feb 11, 2022 at 7:17
Add a comment
|