We need to init (NOT BACKFILL) a data vault from a incremental archive of 10 years daily deltas. The deltas are no real deltas but let’s say that the records have been extracted with some sort of CDC. The daily batches have been preserved and for many satellites the daily number of changes are in the 100.000 rows ballpark, so we don’t want to use a daily loop and consider a rehashing query approach.
In the Migration chapter of the Data Vault Guru it is recommended that we use the load timestamp of when the data is migrated.
In the history we have a technical load timestamp that indicates when the original source file was loaded. Each row also contains a applied timestamp (dv_appts). For performance reasons we think it would be best to partition the data by business years.
So we might have three options for the load timestamp:
-
Use a single load timestamp for the migration over all partitions
-
Use a load timestamp per migration partition and load the partitions sequentially
-
Use the historical source load timestamp
I’m inclined to option 2, but include the original load timestamp as an extra metadata attribute.
Any advise?
Thank you in advance.