ETL process

ETL (Extract, Transform and Load) is a process in data warehousing responsible for pulling data out of the source systems and placing it into a data warehouse. ETL involves the following tasks:

- extracting the data from source systems (SAP, ERP, other oprational systems), data from different source systems is converted into one consolidated data warehouse format which is ready for transformation processing.

- transforming the data may involve the following tasks:

  applying business rules (so-called derivations, e.g., calculating new measures and dimensions),

  cleaning (e.g., mapping NULL to 0 or "Male" to "M" and "Female" to "F" etc.),

  filtering (e.g., selecting only certain columns to load),

  splitting a column into multiple columns and vice versa,

  joining together data from multiple sources (e.g., lookup, merge),

  transposing rows and columns,

  applying any kind of simple or complex data validation (e.g., if the first 3 columns in a row are empty then reject the row from processing)

- loading the data into a data warehouse or data repository other reporting applications