Looking back at the significant data trends over the last 20 years, we have moved from relational databases to data warehouses, data lakes, and now data mesh. We can insert a few more concepts in between, like non-relational databases (NoSQL), data virtualization, the move from on-premises to the cloud, and more. But have we succeeded and made significant progress in managing and mastering the Enterprise data? The results are somewhat mixed.
On paper, data mesh is an attractive idea and makes sense, with the concepts of domain owners, data-as-a-product, self-service, and federated data governance. But implementing it will be challenging and take a long time. Not to say that getting there hundred percent is probably an illusion.
Many data simplification and rationalization projects have delivered too little, not to say they have failed. There are multiple reasons for this. First, let’s recognize that it’s a complicated problem to solve. Second, there is always something more important to do in terms of new critical data requirements. Thirds, data requirements keep evolving, and there is no perfect “master” data model that can handle everything. I’ll stop here as my objective is not to provide an exhaustive list.
Maybe it is time to consider a different data paradigm and approach the problem from a different angle. While we should continue to simplify the data landscape and put better data governance in place – I would definitely push the concept of data mesh – we should also recognize that this will be a very long journey and that some data “mess” will remain for a very long time, if not forever. So why not acknowledge and accept it and link all this data “mess” together via an abstraction layer? And I am NOT talking about data virtualization like Denodo and others would think about it.
This is where the latest advancements in artificial intelligence will play a crucial role. What I am proposing here would not have been possible 5-10 years ago. We need two things: 1) an intelligent engine to connect disparate databases and 2) some generative AI to help the user, whether a human or a machine through APIs, to get the data she needs from these disparate databases. These tools exist today and could be deployed across the Enterprise. I am currently discussing the idea with a Swiss startup, and some proof of concept could work within a few months. A full deployment could also be very fast.
This could revolutionize our thinking about Enterprise data. Stay tuned. I will continue to discuss this topic over the coming weeks and months.
Digital Transformation of Asset Managers and Wealth Managers 