Creation of Generic Data Pipeline 💻

📍 Introduction:

In this post, we will see how to create a generic data pipeline.

📺 Problem Statement:

Suppose we need to pull data from 5 different tables, we can create 5 copy activities. But what if we need to pull data from 100 different tables, creating 100 copy activities is not a feasible solution. Hence we will create a generic data pipeline using ForEach activity.

🔭 Steps to create a Generic Data Pipeline:

1. Create 2 generic data sets, one for the source and the other one for the sink. 🔦

   

   

2. Now create a pipeline with ForEach activity. 🔦

   

3. In the ForEach activity create an array that takes the table name as a parameter. 🔦

   

4. Now create a copy activity inside the ForEach activity. 🔦

   

5. Now in the source data set pass item() as a parameter, this iterates all the tables. 🔦

   

6. Repeat the same activity for the sink data set.🔦

   

7. Now run the pipeline and all the data are copied successfully. 🔦