The Future of Data-Led Organisations: Introducing Data Mesh

We have the opportunity to work with organisations across different industries, witnessing the good, the bad, and the ugly in terms of data process, infrastructure and analytics.

Many companies have not prioritised their data strategy because they have effectively served customers without a focus on data for a long time, making it unclear why they should invest in something with an uncertain return on investment (ROI).

However, there comes a tipping point:

1. The burden of manually extracting insights.
2. The increasing number of data sources.
3. Lack of a clear single source of truth.
4. The feeling of untapped data potential.
5. Concerns about data compliance, security, and breaches.
6. The fear of competitors gaining a competitive advantage.
   
   

These factors often lead to a resounding cry within the organisation:

"We need to do something about our data!"

But where do you start? Is there a right way to begin a data architectural build?

To answer this question, I conducted interviews with Heads of Data over the past few months. By starting with the end goal in mind, we can determine the North Star that organisations should strive for.

Key Finding 1:

The main obstacle in any developed data analytics function is the bottleneck caused by limited data access and skill sets. As the central intelligence grows, more teams seek to use it, necessitating a way to allow data and insights to flow freely.

So, how can organisations achieve this when data is complex and only understood by technical experts?

The solution lies in cultural, technical, and strategic changes. At a high level, organisations need to:

1. Invest in technology that is cloud-based, scalable, and accessible.
2. Build a data-focused team that includes experts in various domains, not just those who are data-savvy.
3. Communicate insights effectively, using a storytelling approach and visualisations.

Key Finding 2:

To establish a modern data environment, a new trend is emerging called Data Mesh, which cannot be purchased off the shelf. Here's a three-point summary of what Data Mesh entails:

1. Decentralised data structure: Domain or product teams take ownership of analytical and operational data.
2. Self-serve infrastructure, data products, and federated governance.
3. A data-driven decision-making culture across the entire organisation.

The benefit of implementing Data Mesh? It doubles analytical throughput. Let's explore each point in more detail.

Data Mesh detail

Decentralised data structure: Instead of a centralised data structure managed solely by a data team, data and knowledge are pushed out to the teams that need insights. This is achieved through domain or product teams that possess extensive knowledge of their respective domains, processes, and data.

They become guardians of the data and products they create. To facilitate this, these teams require a data platform infrastructure that enables self-service and the creation of domain-specific data products.

These products can be made available to other domains for cross-domain analysis. An overarching federated governance ensures adherence to organisational rules and industry regulations.

If you need an example, consider a retailer with both physical and online stores. One key domain would likely be CRM, and the Customer View product associated with it would be valuable to almost all departments/teams.

The CRM domain team has deep knowledge of customer data. Operational data is ingested into the defined Customer View data product, such as a Single Customer View, which addresses 90% of typical queries.

The CRM team can also build their own data products, like customer segmentation, which can be shared with other domains.

Other domains in this example could include Management Support, responsible for delivering dashboards and reports for C-level executives, and Product, focused on monitoring product offerings, stock levels, pricing structures, and more.

Key Finding 3:

Transitioning to a Data Mesh structure takes time to evolve processes, platforms, and people. While the guiding North Star might seem distant for many organisations, there are multiple paths to reach this ideal state. Real-life experiences shared by industry experts can guide us.

1. Start with a centralised data warehouse - Although the end goal is decentralisation, it's essential to take initial steps and establish a centralised foundation.
2. Consider focusing on specific assets that draw people's attention.
3. Keep data engineers and data scientists centralised initially.
4. Gradually push data analysts to teams/departments to facilitate their learning of domain knowledge.
5. Employ new technologies like reverse ETL tools for activation in platforms instead of relying solely on data domain engineers.
6. Once established, gradually move data engineers and data scientists to bolster and evolve domain teams.

To determine where you are on this journey, organisations should assess their own pace of development and create a plan that outlines their current status and future goals.

The Evolution of Data:

Stage 1 - Siloed data: Tools and software are available for various purposes, but integrating data from different sources proves challenging, often requiring complex Excel manipulations to generate reports.

Stage 2 - Business Intelligence 101: To address multiple data sources, organisations employ Business Intelligence tools like Power BI, Tableau, or Qlik. While these tools provide answers, as the data model expands, dashboard performance slows, sometimes necessitating significant waiting time.

Stage 3 - Data warehouse intelligence: Data flows efficiently through the system via APIs, RPAs, and manual files, reaching a data warehouse where analysts can utilise their skills with reporting tools. This is an achievement organisations work towards. However, as the system gains traction, a backlog of data requests for report changes, deep-dive analysis, new data sources, fixes, and issues starts to accumulate. The strain of a centralised system becomes apparent.

Stage 4 - Domain team intelligence: Domain teams are established to answer their own questions, build products, and define new data for others. This approach relieves pressure on the central data team, allowing them to focus on larger projects and advanced analytics.

Stage 5 - Artificial Intelligence (AI): While AI in a decentralised system is uncharted territory, managing constant maintenance and governance can become overwhelming and costly as the system grows. Future advancements in technology and AI may simplify this process.

1. Pick your cloud instance of choice (Azure, GCP, AWS, Snowflake, Databricks)

2. Utilise technology for ingestion (Fivetran, Data Factory, Talend, AWS and Google offerings)

3. And reverse ETL tools (Hightouch, Census, Dataform, Airbyte and Grouparoo)

Seek expert advice to ensure the right technology aligns with your organisations strategy, and build your implementation in stages.

Feel free to reach out to us with comments, questions, suggestions, or simply to have a coffee and chat.