Data product management involves a structured approach to creating, deploying, and maintaining data products that deliver valuable insights and drive business decisions. This article will delve into the phases of data product management, the roles of the Data Product Team and Data Product Consumer, and the supportive role of the Platform Team.
Data product management is the orchestration of data products from conception to implementation and beyond. It involves a meticulous approach to understanding user requirements and designing solutions that not only meet these needs but also comply with industry standards and regulatory frameworks.
By fostering innovation and agility, data product management ensures that organizations can harness the power of data to drive informed decision-making and stay ahead in today's competitive landscape.
Let's start with a few key definitions:
The end-users or stakeholders who consume data products. They rely on the insights and functionalities provided by the data product to make informed decisions and drive business value.
Now, let's cover the phases of this process (also present in the image above).
This first phase involves gathering and defining requirements and purpose for the data product. The Data Product Team collaborates with stakeholders and domain owners to understand their needs and set clear objectives for the data product.
Prototyping involves creating initial versions of the data product to test concepts and validate ideas. This phase is crucial for identifying potential issues early on and refining the product based on feedback. In this phase, only semantic information will be defined with no definitive physical structures. This includes input ports, output ports, description of the business logic, and how the data product fits with the business glossary/ontology.
During the bootstrap phase, the necessary infrastructure and frameworks are established. This sets the foundation for the development and deployment of the data product. All the repositories are created starting from the blueprints provided by the Platform Team. This ensures that teams can start developing while being on the right track.
In this phase, the Data Product Team develops the data product, writing the necessary code and integrating various data sources. This is where the product takes shape.
Documentation is created to provide a clear understanding of the data product’s functionalities, data flows, and usage. This ensures transparency and ease of maintenance. Documentation includes data contract metadata, linking with business terms from the business glossary, and other information that could be needed to drive trust among all the users.
Rigorous testing is conducted to ensure the product meets all requirements and complies with relevant standards and regulations. This phase is critical for maintaining high-quality standards, governance, and interoperability among data products. Computational governance plays a crucial role in this phase.
The product is deployed to the production environment, making it available to the Data Product Consumer. This phase includes setting up the necessary infrastructure of all the Data Product components, application deployment, and publishing of metadata to all the platforms needing them (e.g. Data Catalog, Marketplace, etc.).
Continuous monitoring of the data product is essential to guarantee that all the promises made by the Data Product Team towards the Data Consumers are kept and maintain trust at high levels among the platform participants. This involves tracking key metrics, monitoring data contracts, and setting up alerts and notifications. Typically each data product needs to implement an observability standard to make it easy to understand what is going on inside the data product itself.
Operational tasks are carried out to maintain the data product’s efficiency. This includes routine maintenance, data deletions, restarts, and responding to any operational issues. These activities are performed thanks to control ports that standardize this pattern.
Change management involves handling modifications and updates to the data product, ensuring it continues to meet user needs and adapts to new requirements or environments. Change management also needs to be performed according to defined standards to avoid disrupting the ecosystem and creating a storm of change management in downstream data products.
Effective data product management is vital for leveraging data as a strategic asset. By following a structured lifecycle and leveraging the support of the Platform Team, organizations can ensure their data products are robust, compliant, and valuable.
The collaboration between the Data Product Team, Data Product Consumer, and Platform Team fosters an environment of innovation, efficiency, and continuous improvement.
Data Product Discovery: The consumer, by accessing a Marketplace, can discover which data products exist and provide useful information for its goals.
Data Product Access: Consumers can ask to get access to specific data products, according to overall governance rules and authorization workflows.
The Platform Team plays a pivotal role in supporting the Data Product Team throughout the data product lifecycle. Here’s how they contribute to each phase:
The Platform Team provides standardized blueprints and templates to guide the development and deployment of data products, ensuring consistency, efficiency, and a full-service experience.
They establish and enforce standards for data quality, security, governance, and many other aspects (observability, control, etc.) ensuring compliance across all data products.
The Platform Team automates the deployment process, therefore streamlining it, while ensuring adherence to governance policies.
The platform guarantees that all the data products behave as expected during runtime while monitoring compliance and triggering needed actions.
Effective data product management is vital for leveraging data as a strategic asset. By following a structured lifecycle and leveraging the support of the Platform Team, organizations can ensure their data products are robust, compliant, and valuable.
The collaboration between the Data Product Team, Data Product Consumer, and Platform Team fosters an environment of innovation, efficiency, and continuous improvement.