# How to model repeating IoT data **URL:** \ **Category:** Data Vault 2.0\ **Created:** [May 11, 2023, 9:55am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790 "2023-05-11T09:55:45Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![sosmaf](https://avatars.discourse-cdn.com/v4/letter/s/ed8c4c/32.png) [@sosmaf](https://forum.data-community.org/u/sosmaf)\ **Post date:** [May 11, 2023, 9:55am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/1 "2023-05-11T09:55:45Z")
Hey, How to correctly model IoT data which rows cannot be uniquely identified and cannot detect delta? ![iot_pseudodata](https://europe1.discourse-cdn.com/flex013/uploads/datavaultusergroup/original/1X/7856aa497af5b610d09e22e1b40768b5c5be9439.png) We have rows which are 100% identical, we have otherwise identical rows but one value changing and we have unidentical rows as shown in the example picture. I was thinking about making a link table with the business keys (Hub1, Hub2) and timestamp as dependent child. Link hash key would be HASH(HUB1,HUB2,timestamp) and then there would be a multi active satellite, which links to the link with the same Link hash key HASH(HUB1,HUB2,timestamp) having all the descriptive data and the value column. Some sub sequence can be generated to the MA-SAT but it might not even be required on the snowflake. The data never changes, but same data can come twice. The MA-SAT would then contain the newest rows per Link HK based on the LDTS. Is this correct approach? ---
**Author:** ![Nat](https://avatars.discourse-cdn.com/v4/letter/n/8c91f0/32.png) [@Nat](https://forum.data-community.org/u/Nat)\ **Post date:** [May 11, 2023, 2:05pm UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/2 "2023-05-11T14:05:59Z")
First of all is that data correct or nonsense… Do you know from the sensor manufacturer/configurer what things here mean/is there a meaning to two observations at the precise same time? Is the source sending all the data possible via the API, or are there other fields but you’re not getting them? ---
**Author:** ![AHenning](https://avatars.discourse-cdn.com/v4/letter/a/96bed5/32.png) [@AHenning](https://forum.data-community.org/u/AHenning)\ **Post date:** [May 14, 2023, 4:26am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/3 "2023-05-14T04:26:08Z")
Sounds like that IoT system is pretty crappy delivering pure duplicates. IoT data belongs in a non historized link. My suggestion for the duplicates is to calculate a field in the stage environement and do distinct select when loading the NHL. Ex: New field named NrOfRowsPerBK. Most rows will contain the value 1. But if you have a duplicate this row will contain a value larger than 1. This way you are filtering the data in stage without losing any data and you can easily load your NHL. Good luck! ---
**Author:** ![patrickcuba](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.data-community.org/patrickcuba/32/19_2.png) [@patrickcuba](https://forum.data-community.org/u/patrickcuba)\ **Post date:** [May 14, 2023, 11:52pm UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/4 "2023-05-14T23:52:19Z")
Never ever hash the timestamp, the data should be loaded to a link-satellite with a dependent-child key being the timestamp. As for the identical rows, but then you counter your own statement by saying that aren’t — still a link-satellite. Consider sat-splitting if in fact the identical data is describing the business object and not the transaction between business objects. ---
**Author:** ![sosmaf](https://avatars.discourse-cdn.com/v4/letter/s/ed8c4c/32.png) [@sosmaf](https://forum.data-community.org/u/sosmaf)\ **Post date:** [May 15, 2023, 5:34am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/5 "2023-05-15T05:34:05Z")
The timestamp **should** be hashed as a part of the LINK HK, right? But of course the timestamp field itself is plain and untouched. What comes to the dupes, I need to have a chat about the duplicates with the source provider _again_. ---
**Author:** ![patrickcuba](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.data-community.org/patrickcuba/32/19_2.png) [@patrickcuba](https://forum.data-community.org/u/patrickcuba)\ **Post date:** [May 15, 2023, 5:49am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/6 "2023-05-15T05:49:22Z")
never include timestamps in the hash keys ---
**Author:** ![sosmaf](https://avatars.discourse-cdn.com/v4/letter/s/ed8c4c/32.png) [@sosmaf](https://forum.data-community.org/u/sosmaf)\ **Post date:** [May 15, 2023, 5:59am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/7 "2023-05-15T05:59:27Z")
Hmm, the link HK will then be same for all the records referring HUB1 and HUB2? So, joining the link to the sat causes cartesian product and you don’t really know which of the link’s row is paired with each entry in the sat ---
**Author:** ![patrickcuba](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.data-community.org/patrickcuba/32/19_2.png) [@patrickcuba](https://forum.data-community.org/u/patrickcuba)\ **Post date:** [May 15, 2023, 7:11am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/8 "2023-05-15T07:11:56Z")
If the link-hk is unique in the link how will you ever get a cartesian product by joining the link to the link sat? SQL 101 ---
**Author:** ![sosmaf](https://avatars.discourse-cdn.com/v4/letter/s/ed8c4c/32.png) [@sosmaf](https://forum.data-community.org/u/sosmaf)\ **Post date:** [May 15, 2023, 7:35am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/9 "2023-05-15T07:35:26Z")
I don’t know if I am tripping or if we’re speaking different language or what but the link-hk will **not** be unique if the timestamp is excluded from the HASH. Refer to this picture ![timestamp_hashed](https://europe1.discourse-cdn.com/flex013/uploads/datavaultusergroup/original/1X/c3e972395f1f943b42467606de6b96ec55fad268.png) ---
**Author:** ![patrickcuba](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.data-community.org/patrickcuba/32/19_2.png) [@patrickcuba](https://forum.data-community.org/u/patrickcuba)\ **Post date:** [May 15, 2023, 7:47am UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/10 "2023-05-15T07:47:35Z")
Of course it will… otherwise you’re doing Data Vault wrong. ---
**Author:** ![RhysHanscombe](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.data-community.org/rhyshanscombe/32/394_2.png) [@RhysHanscombe](https://forum.data-community.org/u/RhysHanscombe)\ **Post date:** [December 1, 2025, 2:52pm UTC](https://forum.data-community.org/t/how-to-model-repeating-iot-data/790/11 "2025-12-01T14:52:17Z")