The Data Mesh concept

In recent years, the concept of data mesh has been gaining attention in the tech industry as a new approach to data management. Developed by Zhamak Dehghani, a software architect at ThoughtWorks, data mesh is an architectural approach that aims to decentralize data ownership, promote collaboration between domain experts, and improve data quality, accessibility, and governance.

According to a recent article in Harvard Business Review, “The Data Mesh: A Modern Data Architecture for Business Agility,” data mesh is “a new way to architect data infrastructure that emphasizes decentralization and the distribution of data ownership and architecture responsibilities.” The article also emphasizes the importance of data governance in a data mesh architecture, stating that “governance processes must be agreed upon and enforced across domains to ensure consistency, data privacy, and data protection.”

Implementing a data mesh approach can bring several benefits to organizations. Here are some of the key benefits:

• Improved Data Quality: experts can manage their own data assets, which can lead to better data quality. Domain experts have a deeper understanding of the data and can ensure that it is accurate and relevant.
• Faster Time-to-Insight: faster access to data by breaking down data silos and enabling domain experts to manage their own data assets. This can lead to faster time-to-insight and faster decision-making.
• Increased Agility: organizations can be more agile by breaking down data silos and enabling domain experts to manage their own data assets. This can lead to faster development and deployment of data products and services.
• Better Collaboration: a collaboration and innovation culture can be fostered by enabling domain experts to manage their own data assets and work together to create data products and services.
• Improved Governance: organizations can implement better governance processes by providing a centralized view of data assets and enabling domain experts to manage their own data assets. This can lead to better data privacy, protection, and compliance.
• Scalability: A data mesh architecture can be more scalable than traditional centralized approaches to data management. By breaking down data silos and enabling domain experts to manage their own data assets, organizations can better handle the volume and variety of data they generate and store.

To enable data mesh, organizations need to implement a technical architecture that supports these principles. One of the key technologies that enable data mesh is microservices architecture. Microservices are small, independent services that are designed to perform a specific function. In a data mesh architecture, each domain owns and manages its own data products, which are implemented as microservices. This enables the domain experts to design, develop, and maintain their own data products independently, making them more agile and flexible.

According to a report by Gartner, “microservices provide a good way to break down data silos and enable more agile development and deployment of data products.” The report also emphasizes the importance of event-driven architecture in a data mesh architecture, stating that “event-driven architecture enables a loosely coupled and scalable approach to data integration and processing.”

Another technology that enables data mesh is event-driven architecture (EDA). EDA is a design pattern that focuses on the communication between software components based on events. In a data mesh architecture, events are used to notify other domains of changes in data products. This enables domain experts to be aware of changes in other domains’ data products and to collaborate on new insights or solutions.

Data mesh also requires a data platform that supports data discovery, data lineage, and data governance. A data catalog is a key component of a data platform, which provides a centralized repository of information about data products across domains. It enables domain experts to discover data products and to understand the data lineage and quality of those products. In addition, a data governance framework is required to ensure that data is managed in a consistent and compliant manner across domains.

Zhamak Dehghani, the creator of data mesh, has also emphasized the importance of a data catalog in a data mesh architecture. In an interview with InfoQ, Dehghani stated that “the data catalog is the heart of the data mesh architecture,” and that it “provides a way for different teams to discover the data that is available to them, understand how it can be used, and who is responsible for it.”

There are many benefits to implementing a data mesh architecture. Firstly, by decentralizing data ownership, domain experts become more invested in ensuring the quality of their data, leading to cleaner, more reliable data. Secondly, this approach enables faster decision-making and reduces the time required to make changes to data, making organizations more agile. Thirdly, data mesh encourages a culture of sharing knowledge and expertise, which can lead to more innovative solutions. Finally, data mesh can be easily scaled horizontally, making it ideal for large and complex data ecosystems.

However, implementing data mesh can be challenging. Firstly, it requires significant changes to existing data management processes depending on current maturity and context, which can be complex and time-consuming. Secondly, decentralizing data ownership can lead to increased costs associated with data management. Thirdly, with multiple domains owning their own data, standardization can be more difficult to achieve, leading to potential inconsistencies and errors. Decentralizing data ownership can also lead to the creation of duplicate data, which can be difficult to manage and can lead to confusion.

To successfully implement a data mesh architecture, organizations need to define their domains and identify the data products associated with each domain. They need to establish data ownership by assigning ownership of each data product to the relevant domain experts. Additionally, organizations should create a governance framework that outlines the rules and standards for data management across domains. They should then build data products that are aligned with the needs of each domain and implement the necessary infrastructure to support data mesh, including data storage, security, and access management.

Here are some key success factors for a data mesh to succeed:

• Executive Sponsorship is required for the success of a data mesh implementation program. Senior leaders should be engaged and supportive of the program, and should help to communicate the vision and goals of the program throughout the organization.
• Implementing a data mesh approach requires a cultural change within the organization. Domain experts need to be empowered to manage their own data assets, and a culture of collaboration and innovation needs to be fostered.
• Strong data governance is essential for the success of a data mesh approach. Data quality, privacy, protection, and compliance need to be carefully managed and monitored to ensure that data is used appropriately and effectively.
• Technical excellence is critical for the success of a data mesh approach. The data mesh architecture should be designed with scalability, performance, and reliability in mind, and should be built using modern technologies and best practices.
• Well-defined data domains are essential for the success of a data mesh approach. Data domains should reflect the organization’s business units and processes, and should be managed by domain experts who have a deep understanding of the data.
• Collaboration and communication are critical for the success of a data mesh approach. Domain experts need to work together to develop data products and services, and tools should be provided to enable effective collaboration and communication.
• Measuring success is essential for the success of a data mesh approach. Key performance indicators (KPIs) should be defined, and regular assessments should be conducted to monitor progress and identify areas for improvement.

Implementing a data mesh approach requires a comprehensive program that covers various aspects, including people, process, and technology. Here are some of the key steps involved in a data mesh implementation program:

1. Define the Vision and Goals: The first step in a data mesh implementation program, like any other program is to define the vision and goals. This involves identifying the business objectives, the data problems the organization is trying to solve, and the benefits the organization hopes to achieve.
2. Form a Data Mesh Team: The next step is to form a data mesh team that includes domain experts, data engineers, data scientists, and other stakeholders. The team should be responsible for implementing the data mesh architecture, defining data domains, and managing data assets.
3. Define Data Domains: The data mesh team should define data domains that reflect the organization’s business units and processes. Each data domain should have its own data products and services, and each should be managed by a domain expert. It is rare that data mesh programs are built from scratch and a difficulty at this point is to design, plan and implement the steps from existing organizations to a data domain oriented architecture where the domain teams can work with complementarity on their respective domains and know how responsibilities are redefined along the way.
4. Implement Data Products and Services: The data mesh team should implement data products and services so that domain experts can manage their own data assets. This may involve developing microservices, APIs, pipelines and other tools that enable domain experts to access and manage their own data.
5. Implement Data Governance: The data mesh team should implement data governance processes that ensure data quality, privacy, protection, and compliance. This may involve implementing data lineage, data cataloging, and other tools that enable the organization to manage and govern its data assets. This is where HIPAA, GDPR and more broadly PII (Personally Identifiable Information) management policies should be defined and enforced.
6. Implement Collaboration and Communication Tools: as the data catalogue of products and services will get bigger, it can get fairly difficult to find information. Before reaching this point, the data mesh team should implement collaboration and communication tools that enable domain experts to work together and share knowledge. This may involve implementing wikis, forums, and other tools that enable domain experts to collaborate and innovate.
7. Monitor and Measure Success: The data mesh team should monitor and measure the success of the data mesh implementation program. This may involve defining KPIs, conducting regular assessments, and making adjustments as needed to ensure the program is meeting its goals.
8. Continuous improvement: each context is different, and we all learn faster while doing. While it is a great accelerator to implement recognized best practicies, continuous improvement can help build even more efficient organizations by questioning the models and trial and error.

Data mesh is an innovative approach to data management that requires a technical architecture that supports decentralization, collaboration, and data autonomy. Microservices architecture, event-driven architecture, and a data platform are key technologies that enable data mesh. Implementing data mesh requires significant investment in infrastructure and a culture shift in organizations. By following these principles, organizations can implement data mesh and take advantage of its benefits.

Several companies have already implemented a data mesh approach and have reported positive outcomes:

1. Zalando: a European online fashion and lifestyle platform, implemented a data mesh approach to improve its data agility and scalability. The company created data domains for each of its business units and implemented a governance framework that enabled domain experts to manage their own data assets. As a result, Zalando reported improved data quality, faster time-to-insights, and reduced data silos.
   You can find an informative interview of Dr. Alexander Borek is Head of Data Analytics at Zalando SE here.
2. ThoughtWorks: a global software consultancy, implemented a data mesh approach to improve its data analytics capabilities. The company created data domains for each of its client engagements and implemented a set of data governance and data quality practices. As a result, ThoughtWorks reported improved data quality, faster delivery of insights, and increased collaboration between data teams and business stakeholders.
   Read their approach here.
3. Maersk: a global shipping and logistics company, implemented a data mesh approach to improve its data governance and analytics capabilities. The company created data domains for each of its business units and implemented a governance framework that enabled domain experts to manage their own data assets. As a result, Maersk reported improved data quality, increased collaboration between data teams and business stakeholders, and faster time-to-insights.
4. ABN AMRO Bank: a Dutch bank, implemented a data mesh approach to improve its data governance and analytics capabilities. The company created data domains for each of its business units and implemented a set of data governance and data quality practices. As a result, ABN AMRO Bank reported improved data quality, faster delivery of insights, and increased collaboration between data teams and business stakeholders.

the data mesh architecture offers a modern and innovative approach to data management that decentralizes data ownership and promotes collaboration between domain experts, leading to improved data quality, faster time-to-insight, increased agility, better collaboration, improved governance, and scalability. Implementing this approach requires significant changes to existing data management processes, which can be complex and time-consuming. However, by establishing data ownership, creating a governance framework, and building data products that are aligned with the needs of each domain, organizations can successfully implement a data mesh architecture that is ideal for large and complex data ecosystems. Despite some potential challenges, the benefits of data mesh make it a compelling option for organizations looking to improve their data management capabilities.

NUJUM is well-equipped to help organizations implement a data mesh approach. Our expertise in data architecture, management, and governance can assist with setting up domain-specific data products, establishing cross-functional teams, and implementing a decentralized approach to data ownership and access. Additionally, our experience with data integration and data quality can help ensure that the various data products are consistent, reliable, and interoperable. With NUJUM’s guidance, organizations can successfully adopt a data mesh approach and unlock the full potential of their data assets. Send us an email!

contact.web@nujum.net

Further reading:

Microsoft: What is a data mesh?
Martin Fowler: How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh
Martin Fowler: Data Mesh Principles and Logical Architecture