How can we implement pipelines using LLMs that can consider information from the internet in their responses?

Question

Let us suppose that we are implementing a data processing pipeline based on LLMs and, in some part of the process, we need to search the internet for finding relevant information regarding some topic, in a way that the LLM can use the retrieved information from the internet as context for building its response. How can we do that?

I know that chatGPT and Gemini can do that. But I would like to understand how we can implement that feature using our custom pipelines, using open LLMs, for example.

I think that we can use a custom query string and, in the specific part of our pipeline, we can call a search engine with this string, collecting information from the top sites, an using that as context in the prompt of the LLM. Does this make sense?

Answer 1

What you're seeking is known as Retrieval Augmented Generation (RAG), and there are many open-source projects hosted on GitHub to do just this. The retrieval sources vary depending on the project; some integrate the Google Cloud Search API, others utilize Bing, and some deploy their own live crawlers.

In your case, what you're saying makes perfect sense. You would retrieve the relevant information and incorporate it into your prompt as in-context learning. Additionally, you might be able to fine-tune a language model (LLM or another type) to map your prompts to the query strings for the search engine API.

At a high level, the process involves taking the initial prompt, mapping it to a query string (or vector depending on the database type), and querying it against an external data source, such as a search engine or database. The retrieved information, which may require preprocessing, is then added as contextual augmentation within the original prompt.