VTWM: An Incremental Data Extraction Model Based on Variable Time-Windows

Jia, Weixing and Xu, Yang and Liu, Jie and Wang, Guiling (2020) VTWM: An Incremental Data Extraction Model Based on Variable Time-Windows. EAI Endorsed Transactions on Collaborative Computing. e1. ISSN 2312-8623

[thumbnail of eai.12-6-2020.166291.pdf]
Available under License Creative Commons Attribution No Derivatives.

Download (2MB) | Preview


Continuously extracting and integrating changing data from various heterogeneous systems based on an appropriate data extraction model is the key to data sharing and integration and also the key to building an incremental data warehouse for data analysis. The traditional data capture method based on timestamp changes is plagued with anomalies in the data extraction process, which leads to data extraction failure and affects the efficiency of data extraction. To address the above problems, this paper improves the traditional data capture model based on timestamp increments and proposes VTWM, an incremental data extraction model based on variable time-windows, based on the idea of extracting a small number of duplicate records before removing duplicate values. The model reduces the influence of abnormalities on data extraction, improves the reliability of the traditional data extraction ETL processes, and improves the data extraction efficiency.

Item Type: Article
Uncontrolled Keywords: change data capture, incremental data extraction, timestamp, ETL
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
QA75 Electronic computers. Computer science
T Technology > T Technology (General)
Depositing User: EAI Editor IV
Date Deposited: 15 Jul 2021 11:32
Last Modified: 15 Jul 2021 11:32
URI: https://eprints.eudl.eu/id/eprint/4678

Actions (login required)

View Item
View Item