Case Study: Hybrid Architecture with Data Fabric and Data Mesh

In September, I published a blog on Data Oriented Computing — Architecture Patterns. Within that post, I discussed two hybrid architecture designes of data fabric and data mesh:

1. Lambda design
2. Kappa design

I encountered a notable real-world hybrid architecture case recently, while designing next-generation platform architecture for an open group community. Ultimately, Lambda design was the best fit for this situation as the open group platform needed to deal with real-time data and static data using a consistent approach. How did we come to that conclusion? Details on the case requirements, strategies, and final architecture below.

Requirements

Based on all the parties of the open group community and the market needed, the next generation platform had to fulfill the following requirements:

1. Borderless business interoperability across different domain applications from different parties.
2. Innovative and evolutionary schemas to support federation of different data sources and data consumption.
3. Data governance, security, lineage and integrity support.

Strategies

Based on the community requirements and broader technology trends within the current market, I used the following design strategies:

1. Open and flexible architecture: As markets and technologies change quickly, design the architecture with an open mind to easily adapt to new technologies, extend to cover new business cases, and grow into an open ecosystem.
2. Interoperability: Easily adapt and integrate with domain applications and services (legacy or native) from customer and vendors.
3. Sustainability: Optimize resource usage to minimize impact on operations and the planet.
4. Backward compatibility: Maintain backward compatibility for existing APIs to be less disruptive to the existing platform.
5. AI/ML/Automation: Maintain potential for future AI/ML enhancements to automate business logic (e.g., smart data quality and security enhancements, smart notifications to domain applications for new data types, smart logic to find related domain services/applications for a domain application).

Architecture Proposal

Based on the requirements of the situation and my design strategies, the Lambda hybrid architecture pattern discussed in this blog was ideal. I applied this pattern through the following architecture.

Component Architecture

The top and bottom layer of Domain Applications and Data Platform, as discussed in the previous blog, follow standard design. The middle layer of collaboration space is critical in this case to guarantee data integrity and security across data platform and multiple domain applications.

Collaboration Space

The collaboration space couples the data platform with domain application and services. This can be known under many names, such as the collaboration layer, collaboration space, coupling layer, or anti-corruption layer. For consistency, we will refer to this concept as “collaboration space”.

Collaboration space has three key components:

Registration Service

All domain applications must register themselves to the platform.

The registration can use a JSON formatted configuration file that contains information about the domain application, such as endpoints, types of data to be used and pulled/updated from data platform, filters for the pull, and dependency domain services and applications.

Notification Service

Notification Service notifies domain applications for certain data events, such as when a new version of certain data is available and updated.

Based on the event type, the notification can be broadcast to all registered domain applications, or be specific to certain domain applications.

Notification is initialized by the Collaboration Service. Based on the event type and notification contents, Notification Service queries domain application information based on registration and sends notifications to desired domain applications.

Notifications should be asynchronous, and do not need to wait for domain applications to finish all related updates.

Once the notification has been received, it is up to the application what the domain application does, whether that is a certain business flow, or no-op.

Collaboration Service

Collaboration Service is the core service of Collaboration Space. This component includes all the logic to handle data flow events and functions:

1. Listens to data and system events from data platform
2. Initializes notification to domain applications.
3. Collaborates data across different domain applications when receiving input requests, and triggers data enrichment flow for the staging data to make them trusted before promoting to platform and other applications.
4. Handles domain application sessions and transactions to maintain data integrity across data platforms and domain applications.
5. Manages communications among domain applications.

This architecture design is powerful and can handle the following data scenarios:

Data Management Scenarios

Three kinds of data based on usage and management:

1. Domain Data that a domain application retrieves from the data platform.
2. Transition data that is shared across different domain applications.
3. Data that needs to be saved back to Data Platform and consumed by multiple (other) domain applications.

Future Extensions and Enhancements

We have just discussed an open and flexible architecture that can integrate any new technologies, including:

1. At the Collaboration Space, plugin ML models to provide smart notifications to domain applications for new data types and events from data platforms, or smart logic to find related domain applications for a domain application.
2. At data enrichment processes, plugin smart data quality/security enhancements from ML models.
3. Integrate with IOT devices and real-time data events when community is ready.
4. Capability to integrate with any potential domain applications.

Conclusions

The same theme from my previous blog (Data Oriented Computing — Architecture Patterns) remains true:

In near term, we need to focus on how to smartly using existing architecture patterns together to create sustainable business solutions with automation.

This architecture design showcases what happens with a real-world case study.

As a personal note, I recently left the open group community due to a project change. I have only the best wishes to the community and hope to implement the above designs soon.