Different data modelling techniques suite different architectures. If your data architecture does not support ANSI compliant SQL semantics on a relational database, you should probably consider something other than data vault — and yes, throwing Spark at something doesn’t make the data storage architecture relational, it makes it a band aid, a very expensive one.
Why separate them? Easy. Like the design of Kimball models the design of DV models must take advantage of the OLAP platform’s algorithms underneath the hood, to name a few:
- Nested loops
- Hash-joins (build and probe)
- Data sketches like bloom filters
The hash join is important, it’s what makes joining facts and dimensions so performant, and when I see a customer build “PIT views” I know they are missing this lesson entirely.
We also preach that your data complexities should be shifted as far left as possible, the theme of this article, The OBT Fallacy & Popcorn Analytics | by Patrick Cuba | The Modern Scientist | Medium
It is not documented anywhere that you shouldn’t have business keys in satellite tables, I encourage it. Why do you need to join to anything if you do not need to?
Why have those separate tables?
- Audibility
- Isolating PII
- Tracking true changes
- Solving model / data complexities upfront so analysts don’t need to
Coincidently many of these topics is exactly what I discuss here: Data Vault is Information Mapping | by Patrick Cuba | The Modern Scientist | Mar, 2025 | Medium
And very doable using Iceberg too, Data Vault on Snowflake and Apache Iceberg | by Patrick Cuba | The Modern Scientist | Medium
Why do I blog these? Because it helps with consultation with customers. How many times when you’re explaining these concepts to an audience and at the end of the call they ask a question you already answered at the beginning of the call! Cognitive load! Sometimes, whilst consulting I might even say, “ah, I have a blog for that, here you go!”
We always recommend training and coaching but I think it is a cheat to simply say, “you’re doing it wrong because you never had training” or worse, “pay me money so I can show you how to do it.” Let me tell you, even trainers get it wrong. No where in the training material do they tell you why the structures work as tables, they just tell you to do it. Or worse, stating that there are problem statements like “link-satellites are dead” without providing any evidence of the problems they encountered. Or spend pages discussing the “colours of DV” — what is the value in that?
I have been trained and certified on both Data Vault variances, so I can see plainly when I’m consulting which DV a modeller models just by looking at it.
It is not helpful going round and stating that you have seen many failed data vaults when the common denominator is yourself, I’m sure you have seen success in your hook (or even share when it has not worked). I won’t defend data vault because I know it is not perfect and I know where and (a lot of the times) why pitfalls occur. A common issue I see is, “we want to build a data vault”, I ask “why”. I would be happy to say “no you shouldn’t” and there can be a host of reasons,
- Your architecture does not support relational semantics
- Your team lacks the maturity needed - technical or even business
- You’re not willing to listen to advice I am giving, you hear it, but you’re not following it —and I never say “I told you so” later, why?
- It is led by or owned by data engineers, their focus is on automation, not data modelling.
Check out the comment section of this article, The Death of Hash Keys. An innovation introduced in data vault… | by Patrick Cuba | The Modern Scientist | Feb, 2025 | Medium
I don’t understand why people are still trying to import hash-keys into PowerBI or Tableau!
Where I have seen it work and work quite well is when customers have a clear vision of what their business architecture looks like. That you’re not building a DV as just another modelling technique but as paradigm shift to thinking of structuring data around business needs. i.e. marrying Enterprise Architecture discipline with how the data is structured.
The pitfall I see here is employee churn and knowledge retention, enterprises must invest in the discipline to continue following principles established on how they will build their data vault, trust me, there’s even variances within the two dominant DVs in the wild!
Have you posted your question to Dan’s forum yet?