Search for a command to run...
Articles tagged with #data
📍 Introduction: In this post, we will see how to create a generic data pipeline. 📺 Problem Statement: Suppose we need to pull data from 5 different tables, we can create 5 copy activities. But what if we need to pull data from 100 different tables,...
📍 Introduction: In this post, we will see how to create a generic data set. 📺 Problem Statement: Whenever a pipeline is made a new dataset needs to be created. So if we create a 100 pipeline 100 new data set needs to be created which is not a feasi...
📍 Introduction: In this post, we will see how to create our first data pipeline. Here we will move SQL data to an ADLS storage in CSV format. Steps to create a data pipeline: 👁️🗨️ Create a linked service that will help to set up a connection to ...
📍 Introduction: In this post, we will go through different components of a Data Pipeline. The above diagram depicts a Data Pipeline where data gets transferred from an on-prem DB to a cloud storage (ADLS) in CSV form. Various components in this pip...
What is Azure Data Factory? 🧵 Azure Data Factory is the heart of Azure data engineering. This is specific to Azure. The most important advantage of Azure Data Factory is it is a Low/no-code tool, hence it does not require much programming knowledge....
What are the different ways to access the Storage Account? Access Keys 🗝️ - These are like passwords. The problem with access keys is that it will give the full control of the storage account. (Hence not recommended) SAS Token 🔐 - This will onl...
