Optimised Transformation Algorithm For Hadoop Data Loading in Web ETL Framework

Gupta, Gaurav and Kumar, Neelesh and Chhabra, Indu (2020) Optimised Transformation Algorithm For Hadoop Data Loading in Web ETL Framework. EAI Endorsed Transactions on Scalable Information Systems, 7 (25): e6. ISSN 2032-9407

eai.13-7-2018.160600.pdf - Published Version
Available under License Creative Commons Attribution No Derivatives.

Download (3MB) | Preview


Web ETL unlike conventional ETL framework requires considerable improvements in all the three layers i.e. Extraction, Transformation and Loading due to the inherent nature of web input data. Websites are huge and are unique source of information, out of such huge information available on the websites, finding and analysing the required and relevant data is critical as the data may be foul consisting of redundant data or misspelled. Determining integrated record that stands for identical real world entities in abundant ways is the major problem to be analysed for any database. Hence, Web ETL transformation layer functionality of data transformation becomes mandatory in determining the pertinent information to be examined. Since the data on the web is “very voluminous” hence loading only clean data in data warehouse is necessary for fast processing to achieve accurate result. The present research focuses on data transformation in web ETL framework and proposes a modified technique to employ token wise sentence sorting to remove redundant records from the patent database along with Levenshtein distance used for string matching. Afterwards the cleaned data is transformed and loaded from this staging area to hadoop environment. The integration of proposed transformation technique with hadoop system delimits the constraint of data processing, storage and retrieval of large data structure from conventional data warehouse system.

Item Type: Article
Uncontrolled Keywords: Redundant Data, Data Transformation, Data Loading, Levenshtein Distance Matching, Hadoop
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
QA75 Electronic computers. Computer science
Depositing User: EAI Editor II.
Date Deposited: 08 Oct 2020 13:52
Last Modified: 08 Oct 2020 13:52
URI: https://eprints.eudl.eu/id/eprint/680

Actions (login required)

View Item View Item