“How to create solution architecture of a modern data platform? “
That’s a question an Architect asked me recently in my Masterclass and I wanted to give that person a round of applause 
because that’s a great question!
Something that more people should be asking themselves!
As someone who has ‘crafted’ modern ‘data platforms’ for ‘insurance’, ‘banking’, ‘fintech’, ‘medical device’ companies…
I’m keenly aware of the priorities and outcomes you should explore before creating a modern data platform.
- Outcome: Which business scenarios demand outcome from the data platform and what are the business KPIs of these business scenarios?
– customer journey & experience uplifting…
– alternative scoring model of risk…
– prediction of next best action in automated workflows (e.g. claims processing) - Co-exist with other solutions: How can a modern data platform co-exist with existing data solutions – data warehouse, ETL platform?
– processing, analytics are moving to cloud for scale & economy of scale
– core application data are still hosted on-prem resulting ingestion of data (ETL) from on-prem to cloud - High velocity data: How modern data platform can ingest data with high velocity from modern digital ecosystems e.g.
IoT sensors…
customer clickstream…
cart abandonment…
weather alert…
security incident… - Unstructured data: How can it manage unstructured data in the data value chain – store, search & join with structured data?
voice, video…
data from internet…
text conversations… - Agile enterprise: How can we insert data platform in an event oriented architecture for the enterprise – ingest events (typically high velocity) as well as generate event response (typically alerts)?
real-time feedback…
serving customer at the point of truth… - Process data: How can we process (clean, enrich, transform) data at scale (some type of parallel processing using Hadoop) as well as in-memory processing (e.g.- Spark, can be other licensed commercial products)?
- Cloud: How can you leverage the power of cloud but is interoperable with multi-cloud & hybrid (with on-prem) solution scenarios?
- Governance: How can we make use of data quality assessment components and governance dashboards available on the data platform?
metadata…
provision access to data, self-service…
security…
- Analytics workbench: How can we offer ML solution workbench as a service for data analysts, statisticians and business owners without platform team lifting a finger?
analytics notebook…
save work…
availability of dataset… - Data, Insight as service: How can we expose data and analytics as services/API from data platform that can evolve as data products?
RESTful API…
monetize insights… - Catalog of services: Which cloud components (PaaS services) can be leveraged for fast time to build and optimal RoI?
Azure Paas services becoming industry standard
gcp has sophisticated AI/ML models out of the box
AWS is catching up - Data platform outcome: What platform outcomes are to be delivered by the solution – speed of processing, performance, cost, RoI, easy to use, open data access (API/REST), security, self-service, business continuity? (Business outcomes in Point #1 above)
If Data Platform is customer’s Achilles heel,
- you wish performance to be better…
- you wish more open access of data…
- you wish more empowerment of users…
You start with a number of features in the Data platform MVP and keep iterating.
Remember the ecosystem (cloud service providers, individual solution components) is evolving.
You need to have an open interoperable Data platform to leverage, integrate in a number of business scenarios in the long haul.
This is at the core of how you start building a modern Data platform.