Big Data Fundamentals 🌍

PublishedDecember 20, 2023

I'm Proficient in a variety of Big Data technologies, including AWS, Azure, GCP, Hadoop, Databricks, Pyspark, Sql, Python, Docker, Jenkins, Git/GitHub, Kubernetes, Azure Data Engineering.

What is a big data ? 👁️‍🗨️

There can be various definations of big data but lets consider a simple one. Any data that is characterized by the 5 V's is considered as a big data. 🗺️

Volume - Huge amount of data which any traditional system cannot handle.
Variety - Data can be in any format i.e structured, unstructured or semi structured.
Velocity - The speed at which data is comming and the power to process these data and make business decision.
Veracity - The quality and correctness of data.
Value - The business decision that can be derived from this data.

#data-engineering #python

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Distributed Processing In Map Reduce

Distributed processing in MapReduce may be summarized in three phases: a map phase, a shuffle phase, and a reduce phase. These phases can be overlapped to some degree to improve efficiency. The map step applies the map function to data local to the p...

Jan 16, 20242 min read

HDFS vs Cloud Storage 🪹

In this post we will see the difference between HDFS and Cloud Storage What is cloud object storage? How it is different from HDFS 🥃 Cloud computing makes extensive use of object storage. This has many advantages, including cost, speed, and scalabil...

Jan 11, 20242 min read

Basic Linux Commands That NeedsTo Rememberd 💯

In this post we will see the top 10 linux command that needs to be remembered 1. sudo 🪴 This SuperUserDo is the most important command Linux newbies will use. Every single command that needs root’s permission, need this sudo command. You can use sud...

Jan 7, 20243 min read

Basic Linux Commands That NeedsTo Rememberd 💯

Getting Started with Distributed Processing And MapReduce 🗯️

What is Distributed Processing ? ⛷️ Distributed processing is a phrase used to refer to a variety of computer systems that use more than one computer (or processor) to run an application. This includes parallel processing in which a single computer u...

Jan 1, 20242 min read

Getting Started with Distributed Processing And MapReduce 🗯️

Frequently used Hadoop Commands 💇

In this post we will see the most frequently used hadoop commands hadoop version - prints the Hadoop version hadoop fs –mkdir /path/directory_name - create directory directory in HDFS hadoop fs -ls /path - command to enlist the files and directori...

Dec 28, 20231 min read

Untitled Publication

37 posts

Command Palette

Comments

More from this blog