Table of contents
📍 Introduction:
In this post, we will go through different components of a Data Pipeline.
The above diagram depicts a Data Pipeline where data gets transferred from an on-prem DB to a cloud storage (ADLS) in CSV form.
Various components in this pipeline 🔰
Linked Service - It mostly contains authentication information. (eg - Server Name, Server address, Port No, DB Username, DB password etc) so that it can get connected to the DB/ADLS etc
Data Set - It understands how to read the data from a table or how to convert the table data to a CSV/JSON etc. format
Activity - It is used to perform a certain action/work on the data eg - Copy
The power for this pipeline comes from Integration run time. Integration run time provides the computation power.