Data Mesh: To Mesh or not to Mesh?

Data Mesh is a technology-agnostic distributed data architecture that makes the data discoverable, widely accessible, secure, and interoperable — giving businesses better decision-making power and faster time to value

In today’s fast-paced, data-driven world, business operations are surrounded by a broad spectrum of data. However, the conventional data architecture is facing tremendous challenges in keeping pace with the rapidly changing demands and landscape of the contemporary world. This need gap paved the way for innovations. An advanced technique that can structure data storage, processing, security, and delivery mechanisms in such a manner that it efficiently supports their growth, scalability, and effectiveness throughout the operations of an organization.

Against this background, today we will unveil the answers to 4 most-asked questions about Data Mesh, a pioneering data architecture with promises to solve the aforementioned challenges:

Foremost, what is Data Mesh?

Data Mesh is unquestionably a dynamic and contemporary concept. Director of Emerging Technology and Thought Works and author of Data Mesh, Zhamak Dehghani, defines this term as a state-of-the-art response to the common problem of centralized, monolithic, and slow-moving data platforms.

A Data Mesh is a technology-agnostic distributed data architecture, where data resides with the team that originates or curates it while the relevant data insights are shared throughout the organization. Data Mesh has also been expounded as a set of principles for designing data architecture. The four fundamental principles are as follows:

1. Owned by one but accessible to all: Data Mesh ensures that data is owned by a specific domain (the one which produces or generates it), but the data itself is accessible in a decentralized way. It simply entails that each business segment identifies relevant data that will be handy in other areas, ensuring that data is accessible to everyone.
2. Data is a product, treat it accordingly: Data Mesh treats data as a product that is shared by the domain team. It is comparable to a standard product development process which includes research and experimentation about how a particular product can be used differently. Similarly, teams are responsible for thinking outside the box to predict how other departments, business areas, and end-users can utilize their data.
3. Data is available with necessary due diligence: Data Mesh ensures that quality data is available throughout the organization with due regard to security and governance. Simply put, while data resides with the domain, department, or team, other business segments can also access and apply it to create reports, and much more (depending on their requirements).
4. Data is governed at the source: In the words of Dehghani, this principle can be summarized in three words – Federated Computational Governance. This is a boutique way of governing data that allows interoperability across domains, irrespective of their differences. By doing governance at a decentralized level, users tend to trust the source and are able to navigate data in the Mesh swiftly.

How is Data Mesh different from its counterparts?

Data Mesh allows teams to curate/generate data and create usable data products for other teams. It also makes certain that platform teams can put their efforts into data engineering while data professionals can handle domain-specific data issues. While business data professionals are responsible for the quality and reliability of the data their teams produce, they can take assistance from platform teams in the face of technical glitches.

Apart from that, Data Mesh design is inclined towards business users and requires relatively minor interference from platform teams. This is unlike centralized data teams that are responsible for everything, from data frameworks and access to dealing with data-related requests.

To conclude, Data Mesh or the decentralized architecture encourages each party to excel in their area of expertise. The platform teams need to focus on technology, engineering, and data pipelines, while the data professionals are accountable for ensuring data quality. This holistic approach ensures end-users can perform their tasks by leveraging data insights without investing time in acquiring the results of a custom request.

What are the obstacles a Data Mesh helps to resolve?

Data Mesh equips teams and leadership with a data-driven mindset. It enables teams to access or agree on any given data of critical importance. It also ensures ownership, accountability, and quality of data insights. It resolves problems like difficulty in dealing with data bottlenecks, slow or inefficient data pipelines, and hard-to-scale data ecosystems.

Furthermore, Data Mesh provides infinitely more sources of data. It also enables even large teams to function with optimized data sources and data-driven features. By leveraging Data Mesh, platforms can eliminate cross-collaborative team struggles and support each side’s function. It can also stop any hindrance caused by the lack of domain knowledge. Data Mesh also provides organizations with detailed and thorough data for a myriad of regions and business areas.

What is the method of Data Mesh implementation?

It begins with a shift in thinking. For instance, incept from centralized data ownership to team-based data, from data as a secondary effect to a valuable product for and within an organization, and from a single team managing a massive centralized data repository to domain teams doing the same but with the help of qualified data professionals.

Building a Data Mesh is a straightforward procedure. Start by including a central data stream platform in your foundation. This should consist of store data, independent publishing by a team, and delivery resources as per user demand/requirements. Following this step, make certain that your data sources are iron-clad. In simple words, make sure they have clear owners (originators or curators of the data) as they will be solely responsible for publishing data on the Mesh.

In terms of security, a team-level establishment of data governance is crucial. Ensure that your platform is equipped to support dynamic scheme changes, connects with both owners and user databases, provides data stream previews and lineage images, and can access requests.

The last but vital step is to implement an intuitive, hassle-free, and user-friendly interface for business users. This will develop an efficient unified experience for ensuring users can get their hands on reliable data without facing any last-mile hiccups.

To Mesh or not to Mesh, that is the question

As explained above, Data Mesh is a technique that enables organizations to decentralize their data and provide meaningful insights to use more constructively. By integrating this new-age method, businesses and data platform teams can focus on their areas of expertise. As each team manages data independently, there will be more detailed governance, faster access and delivery, and better compliance with domain-specific/local regulations. In a nutshell, Data Mesh equips businesses to harness the power of their data efficiently, which can automatically assist them in making data-driven decisions and ensure an agile standpoint.

By Dr. Anil Kaul is CEO, Absolutdata (an Infogain company) & Chief AI Officer, Infogain & Harshit Parikh is VP, Global Practice lead, Infogain