Unlocking value from data: How AI agents will impact 2024

If 2023 was the year of generative chatbots and AI-powered search, 2024 was all about AI agents. Devin's start earlier this year grew into a true phenomenon, offering enterprises and individuals a way to transform the way they work at various levels, from programming and development to personal activities such as planning and booking tickets for holidays.

Among these widespread applications, we also saw an increase of data producers this year – AI-powered agents that handle different types of tasks across the data infrastructure stack. Some did basic data integration work while others handled downstream tasks, such as analytics and prospective management, making things simpler and easier for enterprise users.

The benefits were better efficiency and cost savings, which made many wonder: How will things change for data teams in the coming years?

Gen AI agents took over data functions

While agent capabilities have been around for some time, allowing enterprises to automate certain basic tasks, there is an increase AI generation has taken things completely to the next level.

With natural language processing capabilities and the use of gen AI tools, agents can go beyond simple reasoning and response to planning multi-step actions, autonomously interacting with digital systems to complete tasks while they collaborate with agents and other people at the same time. They also learn how to improve their performance over time.

Cognition AI's Devin the first major agent grant, enabling engineering at scale. Then, larger players began to provide more targeted enterprise and personal agents powered by their models.

In a conversation with VentureBeat earlier this year, Google Cloud's Gerrit Kazmaier said he heard from customers that their data users were constantly facing challenges including self- manual work migration for data teams, reduced data pipeline cycle time and simplified data analysis and management. Basically, the teams were short of ideas on how they could create value from their data, but they didn't have time to implement those ideas.

To fix this, Kazmaier explained, Google revamped BigQuery, its core data infrastructure offering, with Gemini AI. The resulting agent capabilities not only enable enterprises to discover, clean and prepare data for downstream applications – breaking down data silos and ensuring quality and consistency – but also supports pipeline management and analysis, freeing up teams to focus on higher value activities.

Multiple enterprises today use Gemini's agent capabilities in BigQuery, including fintech companies more thanwhich enabled Gemini's ability to understand complex data structures to automate their query generation process. A Japanese IT company Alone also uses Gemini SQL generation capabilities in BigQuery to help their data teams deliver insights faster.

However, finding, preparing and assisting with analysis was only the beginning. As the basic models evolved, even granular data operations—initiated by startups specializing in their own fields—were targeted by deeper, agent-driven automation.

For example, AirBite and fast has made headlines in the data integration sector. The first launched a helper that created data connectors from API document links in seconds. At the same time, the latter developed its broader application development offering with agents who created enterprise-level APIs – whether for reading or writing information on any subject – using only natural language descriptions.

Based in San Francisco Final AIfor its part, focusing on various data operations including documentation, testing and transformations, with new DataMates technology, which used agent AI to extract context from the entire data stack. Many other startups, including Redbird and Rapid Canvasare also working in the same direction, saying they offer AI agents that handle up to 90% of the data tasks required in AI and analytics pipelines.

Agents powering RAG and more

In addition to extensive data operations, agent capabilities were also explored in areas such as discovery-augmented generation (RAG) and downstream workflow automation. For example, the team behind the vector database Inflammation recent discussion of the idea RAG agenta process that allows AI agents to access a wide range of tools – such as a web search, calculator or software API (such as Slack / Gmail / CRM) – to retrieve and validate data from multiple sources to increase the accuracy of responses.

Later, towards the end of the year, Snowflake experience revealed, giving enterprises the option to set up data agents that can tap not only business intelligence data stored in their Snowflake instance, but also structured and unstructured data across on siled third-party tools – such as sales transactions in a database, documents in knowledge bases such as SharePoint and information in productivity tools such as Slack, Salesforce and Google Workspace.

With this additional context, the agents display relevant views in response to natural language queries and take specific actions around the generated views. For example, a user could ask their data agent to enter the surface views in an editable format and upload the file to their Google Drive. They could even be encouraged to write to Snowflake tables and make data changes as needed.

Lots more to come

Although we may not have covered every application of data agents seen or announced this year, one thing is very clear: The technology is here to stay. As gen AI models continue to grow, the adoption of AI agents will move at full steam, with most organizations, regardless of sector or size, choosing to delegate repetitive tasks to specific agents . This translates directly into efficiency.

As evidence of this, in a recent survey of 1,100 technical operators who did Capgemini82% of respondents said they plan to integrate AI-based agents across their stacks within the next 3 years – up from 10% currently. More importantly, as many as 70 to 75% of respondents said they would trust an AI agent to analyze and synthesize data on their behalf, as well as handle tasks such as creating and updating -code development.

This agent-driven shift would also mean significant changes to how data teams work. Currently, there are no producer outcomes at the production level, which means that humans will have to take over at some point to fine-tune the work for their needs. However, with a few more advances over the coming years, this gap is likely to close – giving teams AI agents that are faster, more accurate and less prone to human errors. often does.

So, to summarize, it is likely that the roles of data scientists and analysts that we see today will change, with users probably moving to the field of AI management (where they could look to hold on to AI tasks) or higher value tasks that the system may struggle to perform.