Introduction to Azure Data Factory (ADF) and Extract Transform Load (ETL) ☄️

What is Azure Data Factory? 🧵

Azure Data Factory is the heart of Azure data engineering.

This is specific to Azure.

The most important advantage of Azure Data Factory is it is a Low/no-code tool, hence it does not require much programming knowledge.

Azure Data Factory is a workflow/orchestration tool that is mostly used to create Data pipelines.

What is an ETL? 🧶

Extract, transform, and load (ETL) is the process of combining data from multiple sources into a large, central repository called a data warehouse.

Let's understand more about ETL 🎭📒

Companies have multiple projects/applications and these multiple projects/applications have multiple databases.

Data Analyst/ Business Intelligence/ Other teams need to generate multiple reports. To generate such reports these teams will hit the databases and run complex queries. These complex queries take a longer time to get executed. As a result the databases also get slow down.

Moreover, these databases are connected to applications hence the application also gets slowed down.

These databases are known as OLTP. (Online Transaction Processing)

The solution is rather than slowing down the databases we will move these databases to data warehouse and this process is known as ETL.

Now the Data Analyst/ Business Intelligence/ Other teams will connect to the data warehouse run the complex queries and generate reports. Hence this will not affect the databases and application performance will also not slow down.

Most of the time this is a typical Data pipeline structure.