Generative AI in Practice for Data Teams
Generative AI for data teams: how to use LLMs to generate SQL, document pipelines, and speed up analysis — with a critical view of their limitations.
In June 2024, Erathos participated in a highly relevant event organized by the Google Developers Group in partnership with Mindhub | Hub de Inovação and the Xperts Community. The event featured several presentations on AI trends and innovations, as well as discussions on challenges and potential solutions. In this article, we will dive deep into the topic of Generative Artificial Intelligence in Practice, which was presented by Luca Nozzoli, Data Scientist from our Experts team.
First, let's understand the concept of LLMs
We use various AI tools as allies in our daily tasks and projects, with chatbots being the most prominent. But what is actually happening behind the dialogue we have with the computer?
The foundation of the AI system is the LLM (Large Language Model), which is intensively trained on millions of parameters to understand, summarize, generate, or predict text content. Some of the most famous examples out there are OpenAI's ChatGPT, Gemini and PaLM developed by Google, Meta's LLaMA, and Transformer models from Hugging Face.
How do we use an LLM in real-world projects?
The applicability of Artificial Intelligence goes far beyond just chatting with a chatbot and being understood. In fact, today we find LLMs being used in financial analysis at large corporations for contexts like fraud detection, and assisting marketing teams with content generation. Another great example of these systems in action every day is the recommendation engines in entertainment apps like Netflix and Spotify, as well as e-commerce platforms like Amazon. In these cases, the user's individual profile is closely monitored to gather data for building recommendations. Additionally, we can use an LLM through retrieval-augmented generation (RAG), which basically consists of feeding materials such as PDFs to the system so it can assist you with tasks that require knowledge outside of what it was pre-trained on.
But how do we choose the ideal model for our problem?
We know that choosing among these innovative solutions can be confusing, especially regarding what to consider along the way. We have consolidated some of the key decisions that need to be made to guide this selection process, which you can follow in the topics below:
Features
LLMs are extremely capable of understanding text, but with recent development breakthroughs, they now also understand and process images, video, and audio to build analyses. You need to understand what your problem entails and if it introduces limitations to your system selection.
Data Volume
Another crucial point of attention relates to the volume and size of the information that the LLM will have to process, while also considering the frequency of data ingestion and requests.
Performance
Connected to the feature requirement, we need to verify—based on benchmarks built by various companies in the industry—the accuracy and quality of what each LLM executes and delivers, and how significant that is for your specific use case.
Cost
Last but definitely not least, cost is one of the main constraints when selecting an LLM. You need to balance the costs and analyze which model best fits both your problem resolution and the current stage of your business.
How to measure the cost of an LLM
When we feed content into an LLM, the model "breaks" it down into smaller units of context called tokens to understand the input. This tokenization process differs according to the algorithm of each solution. In this case, it is important to consider that depending on the number of tokens to be processed, the cost can vary.
Defining the best system in practice
In this practical example, we will rely on the 4 decision factors presented earlier to compare different versions of Google's Gemini LLM. Imagine you are a Data Scientist at a company that has several business guideline manuals saved in private text files that are not open to the public. Consequently, Gemini does not have access to this information. Your manager wants a tool similar to a ChatBot, where he can query the data within these files quickly and easily. You must build a retrieval-augmented generation (RAG) solution, processing the data internally and feeding it to a closed-scope model (not public-facing) to assist with company onboarding and guidance. Finally, your manager states that he is willing to allocate whatever resources are necessary to get a tool that meets all requirements, with no budget constraints.
Features
Since the focus of this exercise is text files and all three versions support text inputs, this criterion will not be the deciding factor for our selection.
Data Volume
A context window of 1 million tokens is significant and represents a great limit, considering that most other tools on the market set this limit to about half that value. Regarding requests per minute, Gemini 1.5 Flash offers a higher throughput, ensuring more agility.
Performance
Looking at the performance of the models above, we notice that Gemini 1.5 Pro delivers the best performance across all analyzed benchmarks.
Cost
Analyzing the costs in the table above, we can see that the cost per request for Gemini 1.5 Flash is nearly 10x lower than the cost per request for Gemini 1.5 Pro.
In this case, since our budget is not a constraint and we are prioritizing quality and accuracy, the best option would be Gemini 1.5 Pro, considering the benchmarks across the 4 analysis factors presented.
In short, we are always limited by the resources we have at our disposal! Every use case is unique, and limiting factors must be carefully weighed to ensure the solution meets the needs and priorities of your specific goal.