DataBrain - AI framework for analytics

Intro

Historically, data analysts have been the service wing in enterprises, fielding questions from business users for ad hoc data pulls, generating reports, and answering follow-up questions.

They spend 90% of their time on repetitive, manual, grunt work that is waiting to be automated.

Data analysts are dead, long live data analysts!

To solve the ad-hoc requests, every enterprise structures teams in 1 of 2 ways:

- A central team of analysts

- Analysts embedded in each function - eg: financial analyst, sales analyst etc

In both cases, analysts spend most of their time creating dashboards, explaining insights, organizing data, sharing reports and many more manual & repetitive tasks.

Current processes for generating business insights are broken:

Slow & Complex

Require back and forth between data teams and business users

High bandwidth

Insight finding and subsequent storytelling requires high cognitive bandwidth & is extremely repetitive.

Low ROI

Despite repetitive back and forth, business users still don’t find what they need

“With established trust in data, every business user becomes a data analyst”.
Business users will derive complex insights in < 3 seconds, win more customers, impact top line and make every enterprise a growth company.

and

"With a data studio and eval framework, every data analyst becomes a data scientist”.
Data Analysts will get to work on high bandwidth, high ROI tasks and impact bottom line directly.

‍

DataBrain’s AI powered analytics suite is purpose built to help enterprises implement LLM’s on top of analytical data - with enterprise guardrails and transparency, so data teams can confidently democratize access to data, cut ad-hoc reporting by 90% and free up data teams to work on high ROI tasks.

What stops enterprises from deploying AI on data today?

Every CIO and head of data is currently strategizing about Gen AI for data. Yet there is a massive gap between a demo-able MVP and building a fully productionized AI analytics system.

Current systems are broken:

Fragmented toolchain

Toolchain purpose built for training LLM’s on analytics doesn’t exist today

High Risk

LLM outcomes are not explainable and an eval framework for observability doesn’t exist

Low reliability

Without a framework, LLM’s are susceptible to hallucinations which is fatal for analytics

The missing piece to unlocking AI powered analytics

The modern data stack has made rapid advancements with the rise of the semantic layer from dbt, advancements in data storage with the adoption of interoperable data formats like Apache Parquet and Iceberg, data lakehouse adoption, and in-memory OLAP databases like DuckDB.

All these tools help clean, store, and serve data with low latency, yet they don’t make data ready for LLM consumption.

Data Prep layer

A significant amount of unstructured and structured data sits in data warehouses and data lakes. This data needs to be modeled and semantically mapped to business terms, with established ontologies. This process requires a deep understanding of graph technologies and data storage. Proper data preparation ensures that the data is in a format that LLMs can understand and utilize effectively.

Guardrails layer

What good is data if it is not secure to share with vendors, customers, and internal teams? Implementing row-level security, hiding columns, and establishing data governance rules are even more crucial when training an LLM. These guardrails ensure that sensitive information is protected and that data sharing complies with regulatory standards.

Eval layer

When training an enterprise-grade LLM, it is essential to understand its accuracy and performance, and watch it improve over time. An evaluation framework is needed to measure the LLM’s effectiveness in handling real-world data scenarios with synthetic data. This framework should include metrics for precision, recall, and overall model robustness. Continuous evaluation and fine-tuning are necessary to maintain high performance and relevance. It is also important to compare and contrast against different LLM providers.

RLHF layer

Incorporating RLHF is critical in training LLMs. RLHF here involves data analysts providing feedback on model outputs for business user generated questions, which is then used to refine and improve the model. This iterative process helps in aligning the LLM with specific business goals and ensures that the AI system remains responsive to user needs and expectations.

Consumption layer

The final step is ensuring that the LLM can be easily consumed by business users. This involves creating user-friendly interfaces and APIs that allow seamless integration with existing business workflows - chat, open ended search like Google, auto summarization of insights and much more. These insights should al The LLM should be able to generate insights, answer queries, and automate routine tasks, thereby freeing up data analysts to focus on more strategic activities.

Product

DataBrain is designed for data analysts, data scientists, and business intelligence teams within enterprises. Our product offering includes:

- Data Studio for data prep: A comprehensive tool for preparing, modeling, and semantically mapping data to business terms, making it ready for LLM consumption.

- Guardrails for enterprise-grade security: Advanced security features including row-level security, column masking, and data governance rules to ensure safe data sharing and compliance.

- Evals to reduce hallucinations: A robust evaluation framework that continuously measures and improves the accuracy and performance of LLMs using real-world and synthetic data.

- Feedback mechanism for RLHF: Tools for incorporating user feedback, allowing data analysts to refine model outputs based on business terms and increase model accuracy non-linearly.

Conclusion

When spreadsheets were invented, productivity increased a hundredfold, sparking a global movement to up skill. Tasks that once took hours or days were being completed in minutes. Initial skepticism about the reliability of spreadsheet calculations eventually faded, and spreadsheets became the de facto tool for anything involving numbers.

Similarly, AI analytics frameworks will transform data analysts from manual task executors to strategic decision-makers. Business users will gain the ability to derive complex insights in seconds. As AI frameworks mature, they will offer high reliability and explainability, much like the evolution of spreadsheet technology.

The future holds a landscape where data teams can focus on high-impact, high-ROI activities. This shift will not only enhance productivity but also reshape how teams are hired, ensuring that data analytics remains at the forefront of business intelligence and strategic planning.

The next era of data analysis is here, and it promises to be as transformative as the advent of spreadsheets, driving tangible improvements in efficiency and business outcomes.

‍