What is a Data Platform?

In the age of generative AI, many people forget that the humble data platform is the foundation for information transformation in today's organizations.

The data platform is how an organization transforms information through knowledge work to achieve goals, generate revenue, and improve cash flow. The data platform is the tool organizations rely on to be more efficient with resources and productive to compete for growth in their market.

Data platforms serve as essential infrastructure for organizations, much like historical innovations such as aqueducts and city codes that enabled societies to function efficiently. When effectively implemented, data platforms act as the operating system of the business, driving workflow acceleration and organizational coordination. They enable faster and more efficient execution of critical business processes, such as claims processing and actuarial analysis, turning information into a strategic advantage rather than a cost burden.

The Role of the Data Platform in an Organization

Technology has had a key role in changing labor for thousands of years. Industries and professions with heavy use "knowledge work" process information that gives their organizations advantages over other competitors that do not use technology as effectively as they do. The better these organizations process information, the more competitive they operate in their market. Today data platforms coordinate and accelerate people, teams, and businesses beyond what can be done by manual knowledge work labor alone, making organizations more competitive. When a data platform is running efficiently, it enables a business to grow at its full market growth potential through minimal information transformation friction.

When data platforms perform poorly, they can create headache, distractions, and extra work for an organization; This slows down all information-related tasks, making the organization less competitive in today's markets. It is imperative that an organization be able to trust the accuracy of the information they work with and be able to access new data at a latency that does not slow down their knowledge workflows.

A well-structured data platform directly enhances an organization’s ability to translate knowledge work into improved cash flow, ensuring financial stability and enabling sustainable growth. By optimizing the management of critical functions such as billing, underwriting, claims, and financial operations, data platforms enhance revenue collection, improve forecasting accuracy, and strengthen liquidity planning. This efficiency not only reduces operational overhead but also enables the company to reinvest strategically in sales, marketing, and new business opportunities without overextending its capital—positioning the organization for sustained expansion and long-term competitiveness.

While data platforms, technology, and AI significantly enhance decision-making, they do not replace the need for human oversight. Critical judgment, strategic thinking, and contextual understanding remain essential, particularly in complex and high-stakes scenarios. However, these systems play a crucial role in accelerating decision-making by eliminating weak options early, running simulations at scale, and providing faster access to relevant information. By augmenting rather than replacing human expertise, data platforms help organizations make more informed, efficient, and effective business decisions.

Ultimately a data platform has to support information obligations within an organization while minimizing processing friction.

How a Data Platform Supports the Line of Business

There are 3 key functions a data platform performs in supporting an organization's information processing workflows:

1. Gather information across internal and external systems
2. Organize information in a usable way
3. Support the information obligations of each business unit

As companies grow, so do both number of internal systems and number of external partners. After a certain point, a knowledge worker can be overwhelmed with the demands of data collection and integration before they can get to their actual core job. This scenario tends to slowly happen in parallel with the organic growth of the business. This is the reason that companies experience the need to re-architect their data platforms after long periods of growth, because the platform no longer supports the information obligations of the company.

With a better platform architecture an organization seeks to lower the information processing friction such that the knowledge workers do less data engineering and more work that has direct customer and revenue impact. Additionally, a data platform should provide information that "just works" with the workflow tools that already exist in an organization.

At Patterson Consulting, we define a data platform through three logical layers:

1. The knowledge work layer
2. The information layer
3. The infrastructure layer

These logical layers map to the functionality needed from a data platform to support core knowledge work in the line of business. Now that we've baselined data platforms with both core functional goals and a logical architecture, let's move on and see how they are physically implemented.

Physical Architecture of a Data Platform

While there are many things to consider when operating a data platform (security, authorization, encryption, observability, etc), a data platform has the following key areas it needs to support at the physical layer:

• Data Integration: pull different data sources (internal and external) together to be combined and analyzed
• Transformation and Orchestration: run all the workflows needed to produce the data models in the information layer
• Information Architecture: arrange the processed data into logical information units and areas that can be easily accessed by the knowledge work layer

While supporting the above functions, the platform also needs to produce valid, complete, and timely data. These functional data model requirements can change the course of infrastructure design for specific cases.

In the generalized data platform architecture diagram below, you can see how we are mapping the logical knowledge work architecture layers (knowledge work layer, information layer, infrastructure layer) over the physical implmentation components of the data platform.

You can see both external data integration and internal data integration on the left side feeding into the infrastructure layer. The blue box that overlaps both the infrastructure layer and the information layer represents the physical layout of the information architecture (e.g., "medallion data pattern") for the platform.

Data Integration

Data integration is the process of combining data from different sources to provide a unified and comprehensive view. This process involves gathering, transforming, and consolidating data from various databases, applications, or systems, often across different formats or structures, to make it consistent and usable for analysis, reporting, or operational tasks.

Information Architecture

The information layer contains the "gold" datasets; these are data models that represent a fully integrated dataset containing valid and complete data from the disparate physical systems it was integrated from.

Information transformation efficiency directly impacts how well a company converts information into cashflow. Out-dated architectures and poorly integrated systems both will slow down information transformation in your organization and will definitely cost you money and market-share.

Common methods in this part of your implementation will be SQL scripts that transform incoming data into information or Python scripts that are crunching files that are being sent to your organization. Many times it will involve multiple platforms and multiple types of languages (e.g., SQL, Python, etc) and will need to be orchestrated by a tool such as DBT.

Data Analysts and Data Engineers are typically the roles you'll see operating in this area of your data platform.

Supporting Information Obligations

The green box on the right side represents the knowledge work layer. This box only contains logical line of business divisions that do the critical work of the business (e.g., "actuary", "claims"). This area contains tooling such as Excel, PowerBI, and other basic reporting tools that are used to do the core line of business activities. The knowledge work layer has a contract with only the information layer, and should never need to go access physical data sources.

A good information layer architecture supports line of business workflows with minimal integration friction. This means the line of business is able to easily use their preferred tools (e.g., Excel, PowerBI, Tableau, GraphQL, SQL, Python) to do their work without getting involved in data integration or data engineering work. A great measure of a data platform is how well it allows the line of business knowledge worker to focus on their core work without having to do extra work to pre-process the information they need.

The Evolution of Information Obligations