GDPR and Data Modeling: Our Journey
Annemie Van Cauter and Rebekka Moyson, Aveve, and Ivan Schotsmans, BI-Community
Data projects are complex. The question is how to combat this complexity without sacrificing functionality?
Most professionals working in the data / analytics/ BI field of work have one of two reflexes:
- Narrowing down the functionality into a single narrative and putting all requests not fitting this narrative out of scope.
- Creating complex solutions for complex problems, entertaining all possible deviations and what-if’s.
Neither reflex is sustainable. This results in projects replacing the results of previous projects instead of amending the data landscape.
In order to battle complexity and create sustainable ecosystems of data related solutions, you need to simply things.
Change will happen and is unpredictable. That’s a given, prepare for it. The question is how to deal with this?
Our answer: continuous reduction of complexity.
How to achieve continuous reduction of complexity?
- ‘Just enough’ principle: take iterative, small steps forward, just enough to allow anyone involved to keep up. This gives you the ability to adapt quickly to changes in the information demand. Do not lose sight of the human dimension. If users are overwhelmed by your megalomane ‘prepared for every possible deviation’ solution, they will simply ignore it, because they don’t understand it. ‘Just enough’ has two sides:
- ’Ask just enough’: A lot of the complexity in data projects stem from the inability of users to precisely specify what they need. They will ask for ‘everything’ just to be sure they will get something they can use. Users need to be taught to ask for small simple steps, learn from using it, give feedback and build out from experience.
- ’Deliver just enough’: Don’t do more than being asked for in terms of content and in terms of functionality. Developers are not responsible for applying the information delivered, so they cannot asses what is needed. Rather, help users to develop their data skills without baffling them with IT concerns.
- Use case approach: create solutions based upon use cases that deliver value. Use cases are validated and prioritized by the user community before being put into development. To avoid the pitfall of expanding solutions haphazard with every use case, you can bundle use cases into principal use cases, representing use patterns of information. You can architect the cohesion between these patterns beforehand and fill the data landscape in time through use cases.
- Decomposition: Complexity is inherent. This makes both data modelling (or rather, translating the logical model into physical implementations) and data processing a tough challenge. And yet we try to cram it all into the ‘one size fits all’ solutions. When faced with a complex solution, decompose it into smaller pieces that work together. Barry Devlin’s REAL architecture is an example of such an approach. How to do this?
- There is an inherent schizophrenia in requirements: it has to be real time, completely unambiguous, highly available and agile in both use and development at the same time. That is simply not possible. Break it up in partial solutions that either meet the high availability / unambiguity demand or the flexibility demand and demarcate the responsibilities of each solution clearly. The principal use case based architecture has drawn some of the demarcation lines upfront.
- Let the partial solutions cooperate and deliver information through master data for intelligence and master data for authorization solutions to create loosely coupled solutions in the data landscape. This ties the use cases together and prevents you from reinvention with every new use case added.
- Separate the ‘as is’ from the ‘as interpreted’: a layered or stacked approach to your data model allows you to create several layers of different speeds of change. When confronted with small steps sidewards or backwards in the ‘just enough’ approach, you can easily modify the model. Most of the time the changes are in the ‘as interpreted’ layer.
Isn’t this all very complex to achieve?
Of course, you need to organise for this to work. But from our experience, clearly demarcating the responsibilities and boundaries of autonomous teams working on partial solutions and aligning them through MDI and MDA interface agreements and an architecture decision tree helps organizations to build and expand large data landscapes without drowning in the complexity of coordinating it all.
Annemie Van Cauter and Rebekka Moyson, joined Aveve as young potentials from Exellys (a local IT talent integrator). Annemie is a young graduate from the University of Leuven with a master in Business Administration as well as one in Information Management. After having spent a vast amount of time abroad in countries like the US, Mexico and South Korea, she recently joined Aveve Group in her hometown as a Business Architect Master Data.
Rebekka Moyson is currently working as a business intelligence architect. This is her first work experience after graduating as a Master of Science in Mathematics. Her objective is to become an authority in the domain of data science.
Their first time to visit to DMZ in 2017 made a lasting impression on them and they look forward to contributing this year.
Ivan Schotsmans is principal and founder of BI-Community.org. He has more than twenty-five years of information management experience in various industries. Throughout his career Ivan has focused on providing straightforward solutions to business and technical problems for International companies with a focus on data warehousing, business intelligence and information quality. He is recognized as subject matter expert in data modeling, information quality and agile business intelligence. Ivan is also (co-)founder and active member of several global organizations (TDWI Benelux Chapter, DAMA, IAIDQ, among others) and for two years he acted as Global Director for IAIDQ. Ivan frequently speaks at information management industry conferences and teaches on graphical facilitation, data warehousing, data modeling and new information management trends.