Generative AI in Practice
Generative AI for data teams: how to use LLMs to generate SQL, document pipelines, and speed up analysis — with a critical view of their limitations.
In June, Erathos took part in a highly relevant event organized by the Google Developers Group in partnership with Mindhub | Innovation Hub and the Xperts Community. There were several talks on AI trends and innovations, as well as discussions about challenges and possible solutions. In this article, we will dive deeper into the topic of Generative Artificial Intelligence in Practice, developed by Data Scientist Luca Nozzoli from our Experts team in his presentation.
First, let’s understand the concept of LLM
We use several AI tools as allies to carry out day-to-day tasks and projects, with special emphasis on chatbots. But what is really behind the dialogue we have with the computer?
The foundation of the Artificial Intelligence system is LLMs (Large Language Models), which are intensively trained with millions of parameters so they are able to understand, summarize, generate, or predict text content. Some of the most well-known examples are OpenAI’s ChatGPT, Gemini and PaLM developed by Google, Meta’s LLaMA, and Hugging Face’s Transformer models.
How do we use an LLM in real-life projects?
The applicability of Artificial Intelligence goes far beyond being able to talk to a chatbot and be understood. In reality, we currently find LLMs being used in financial analysis at large corporations in contexts such as fraud detection, and we also see them helping marketing teams generate content. Another major example of these systems operating in daily life is recommendation engines in entertainment apps such as Netflix and Spotify, as well as in product e-commerce apps like Amazon; in these cases, each user’s individual profile is closely tracked to collect information for building recommendations. In addition, we can use an LLM through assisted generation, which basically consists of providing materials such as PDFs so the system can help you accomplish something that requires knowledge beyond what it was originally given access to.
But after all, how do we choose the ideal model for our problem?
We know the selection process among these innovative solutions can be confusing, especially regarding what we should consider along the way. We consolidated some of the main decisions that should be made in order to carry out this selection—follow the topics below:
Features
LLMs are extremely capable of understanding text, but with development advances, today they can also understand and translate images, videos, and audio for analysis. It is necessary to understand what your problem involves and whether it implies constraints on system choice.
Data Volume
Another crucial point of attention is related to the quantity and size of the information that will have to be processed by the LLM, also considering data submission frequency.
Performance
Connected to the features question, we need to check—according to benchmarks built by several companies in the market—the accuracy and quality of what each LLM executes and delivers, and how significant that is for your problem.
Cost
Last, but clearly not least, we have cost as one of the main restrictive factors for selecting an LLM. You need to weigh the numbers and analyze which one best fits problem-solving needs and the current business moment.
How to measure the cost of an LLM
When we provide some form of content to an LLM, to understand what was presented, it “breaks” it into pieces of context, which we call tokens. This context tokenization happens differently according to each solution’s algorithm. In this case, it is important to consider that depending on the number of tokens to be processed, the cost may vary.
Defining the best system in practice
In this practical example, we will rely on the 4 decision factors presented earlier and compare different versions of Gemini’s LLM. Consider that you are a Data Scientist at a company that has several business-guideline manuals saved in privately owned text files, not open to the public. In this sense, Gemini does not have access to this information. Your manager would like to have a tool similar to a chatbot, where they could quickly and easily query the data contained in those various files. You must therefore build an assisted-generation solution, processing internal data and feeding it to a closed-scope model (not open to the public) so it can help with understanding and guidance within the company. Finally, your manager states they are willing to allocate the necessary resources to obtain a tool that meets all requirements, with no cost constraints.
Feature
Considering that the focus of the exercise is text files and all three versions handle text inputs, this criterion will not be used for selection.
Data Volume
An amount of 1 million tokens is significant and a good defined limit, given that most other tools on the market set this limit at around half that value. Regarding requests per minute, Gemini 1.5 Flash presents a higher number and ensures more agility.
Performance
Looking at the performance of the models above, we notice that Gemini 1.5 Pro has the best performance across all analyzed criteria.
Cost
Analyzing costs, in the table above we can see that the cost per request of Gemini 1.5 Flash is almost 10x lower than the cost per request of Gemini 1.5 Pro.
In this case, since our budget indicates no constraints and we are prioritizing quality and accuracy, the best option would be Gemini 1.5 Pro, considering benchmark results across the 4 analysis factors presented.
In summary, we are limited by the resources we have at our disposal! Each case is unique, and limiting factors must be considered carefully to ensure the solution meets the needs and priorities of your specific objective.